Deploying Docker Hub Image to Elastic Beanstalk: i/o timeout

By richardtylee

A few months back we saw the following error while trying to deploy a Docker image to AWS Elastic Beanstalk:

Command failed on instance. Return code: 1 Output: 
[CMD-AppDeploy/AppDeployStage0/AppDeployPreHook/] command failed 
with error code 1: /opt/elasticbeanstalk/hooks/appdeploy/pre/ P
ulling repository username/repo 2015/10/23 22:50:35 Get read tcp 
i/o timeout Failed to pull Docker image username/repo:latest: Pulling repository 
username/repo 2015/10/23 22:50:35 Get read tcp 
i/o timeout. Check snapshot logs for details..

At first, it happened intermittently; maybe a third of the time.  After time, it occurred more than two-thirds of the time.  This ended up bottlenecking our QA team.

Since it was failing on docker pull, we originally thought this was due to our docker images being too big. We spent time to optimize our Dockerfile and reduce the number of layers created during a build. This seemed like a worked for a while, but we started seeing it again after a few days.

Looking closer at the error message, we see that it fails specifically on this call: read tcp 
i/o timeout. Check snapshot logs for details..

The call to the tags endpoint lists all the tags.  Did we have so many tags in the repo that listing them would cause a timeout most of the time?  To test theory, we create a new repo, uploaded our most recent image and deployed to Elastic Beanstalk.  Success!

We reached to Docker for help.  There are a couple of options to work around this:

  1. Docker 1.8.x doesn't have this issue.  This doesn't work for Elastic Beanstalk (which currently supports up to Docker 1.7 at the time of this writing).  I haven't verified this on other hosting platforms, but this would have been the easiest fix.
  2. Delete or create a new repo.  This is a hack, but we chose to do this initially as our team was bottlenecked.
  3. Delete unused tags.  Docker just re-added this functionality in Docker Hub (January 2016).  Since we deploy often, we create many development and staging specific tags.  These can cleaned up afterwards.  We only need to keep the last few master tags, in case we want to rollback.