Posts tagged “Docker”

Deploying Docker Hub Image to Elastic Beanstalk: i/o timeout

By richardtylee.

A few months back we saw the following error while trying to deploy a Docker image to AWS Elastic Beanstalk:


Command failed on instance. Return code: 1 Output: 
[CMD-AppDeploy/AppDeployStage0/AppDeployPreHook/03build.sh] command failed 
with error code 1: /opt/elasticbeanstalk/hooks/appdeploy/pre/03build.sh P
ulling repository username/repo 2015/10/23 22:50:35 Get 
https://registry-1.docker.io/v1/repositories/username/repo/tags: read tcp 10.0.0.1:443: 
i/o timeout Failed to pull Docker image username/repo:latest: Pulling repository 
username/repo 2015/10/23 22:50:35 Get 
https://registry-1.docker.io/v1/repositories/username/repo/tags: read tcp 10.0.0.1:443: 
i/o timeout. Check snapshot logs for details..

At first, it happened intermittently; maybe a third of the time.  After time, it occurred more than two-thirds of the time.  This ended up bottlenecking our QA team.

Preventing PID 1 Zombie Reaping Problem in Docker

By richardtylee.

Our team had been seeing instability in many of our Docker environments on ElasticBeanstalk.  This usually meant we had to rebuild our environments to get it working again.  While researching possible causes, we came across a post about the PID 1 Zombie Reaping problem.  I won't be going into detail on why this is a problem as the post covers it pretty thoroughly.  Here was our problem: on deploys, zombie processes gets left behind when we kill a container's process to start a new one.

To resolve this issue, we must understand the difference between eval and exec.  In eval, the process spawns a child process; in exec, you stay in the same process, which is what we want.

The CMD instruction in the Dockerfile accepts both formats.


CMD command param1 param2

does eval


CMD ["command","param1","param2"]

does exec

Reference to the CMD instruction can be found here.

An important note is that this is not a complete solution to our problem.  If command was a start script, we need to also exec the final command in that script.  Otherwise, you will get stuck in the script.

For example, if the start script ended with:


unicorn -c docker/config/unicorn.rb

it would need to be changed to:


exec unicorn -c docker/config/unicorn.rb

Thanks to Tung, Benson and Eddie for explaining this to me.

How to Make Docker Images Smaller: Introduction

By richardtylee.

Docker has become very popular in the last few years.  At work, we've been using it for over a year now.  As we develop, images for our apps are becoming very large. This is becoming an issue because building, pulling and pushing these images are taking longer and longer. Furthermore, our deployment process has been intermittently failing when doing docker push to Docker Hub; this happens particularly often when upstream internet is slow. Our team tried a handful of different techniques to shrink our image sizes.

This week, I'll go over the different techniques to shrink Docker images. We'll look at the advantages and disadvantages.  Depending on your engineering organization and architecture, some will worker better than others.

  1. Base OS Images
  2. Ruby and Rails Images
  3. App Code Separation
  4. Docker Squash

How to Make Docker Images Smaller: Base OS Images

By richardtylee.

Building a base image starts with choosing a Linux distribution image. There are quite a few options:

  • gliderlabs/alpine is by far the smallest and minimalist. Google search: 499,000 results.
  • debian is used by the Docker Ruby and Rails community, which it will require the least maintenance. Google search: 32,900,000 results.
  • ubuntu is used by the Docker populous at large, which means a large community. Google search: 65,400,000 results.

REPOSITORY          TAG           VIRTUAL SIZE
gliderlabs/alpine   latest        5.03 MB
debian              latest        84.98 MB
ubuntu              latest        188.3 MB
centos              latest        210 MB

Our team looked at a number of factors before choosing debian as our base. As ruby is our primary development language, we wanted to base our images on a distro that had strong community support. Debian also provides tooling that our dev ops team is familiar with.

There have been a few shops that are using alpine in production: https://blog.codeship.com/build-minimal-docker-container-ruby-apps/

While alpine isn't something we'll be using, it may be worth revisiting in the future as adoption picks up.

UPDATE: (03/24/2016) Docker just announced that they build their new engine on Alpine Linux.  There looks to be some momentum building for Alpine.

How to Make Docker Images Smaller: Ruby and Rails Images

By richardtylee.

As I stated earlier, we are primarily a Ruby on Rails organization.  As such, there are plenty of ruby and rails community-supported docker images available to used as base images.

We took a look at some of the ruby and rails images we use. All these images are debian based. Even amongst minor versions, we found large differences in size. In general (though not always the case), the latest minor version will be the smallest size. Since minor versions are backwards compatible, we should be on the latest minor version. It is a good idea to standardize the ruby and rails version we use for our services. Unless there's a specific reason to deviate, we should sit to one version for consistency.

Below is are some of the versions I was able to download and their sizes:

How to Make Docker Images Smaller: docker-squash

By richardtylee.

docker-squash is a Go app that merges layers to make images smaller. The savings using docker-squash depends on how many layers it can effectively squash, so mileage may vary.

Let's squash richardtylee/railsapp and tag it as squashed:


REPOSITORY                      TAG                             VIRTUAL SIZE
richardtylee/railsapp squashed 1.119 GB
richardtylee/railsapp latest 1.141 GB

So we can save about 22 MB.