Posts tagged “Devops”

Chef::Exceptions::ChildConvergeError While Deploying to OpsWorks

By richardtylee.

I attempted to deploy my Rails app to AWS OpsWorks and get the following error:


[2016-01-01T08:39:36+00:00] ERROR: Running exception handlers
[2016-01-01T08:39:36+00:00] ERROR: Exception handlers complete
[2016-01-01T08:39:36+00:00] FATAL: Stacktrace dumped to /var/lib/aws/opsworks/cache.stage2/chef-stacktrace.out
[2016-01-01T08:39:36+00:00] ERROR: deploy[/srv/www/myapp] (deploy::rails line 65) had an error: Chef::Exceptions::Exec: if [ -f Gemfile ]; then echo 'OpsWorks: Gemfile found - running migration with bundle exec' && /usr/local/bin/bundle exec /usr/local/bin/rake db:migrate; else echo 'OpsWorks: no Gemfile - running plain migrations' && /usr/local/bin/rake db:migrate; fi returned 1, expected 0
[2016-01-01T08:39:36+00:00] FATAL: Chef::Exceptions::ChildConvergeError: Chef run process exited unsuccessfully (exit code 1)

OpsWorks doesn’t give the best logging from the AWS Console. All we know from this is that the deployment failed on the following line:


if [ -f Gemfile ]; then echo 'OpsWorks: Gemfile found - running migration with bundle exec' && /usr/local/bin/bundle exec /usr/local/bin/rake db:migrate; else echo 'OpsWorks: no Gemfile - running plain migrations' && /usr/local/bin/rake db:migrate; fi

This means that it failed during a migration, but several possible issues could have caused that call to fail. As John C. Bland suggests, the best option to SSH into the EC2 instance and get the stacktrace.

If you haven’t set up ssh access to your instances, you can read: http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-key-pairs.html

When I ssh’ed into my instance and tried to run /usr/local/bin/rake db:migrate, I found a handful of problems. These are just a few that I encountered:

Scale Down to Zero on Elastic Beanstalk

By richardtylee.

AWS Elastic Beanstalk simplifies application management, with server scaling out-of-the-box.  Scaling is pretty idiot-proof.  In fact, if you try to scale down to zero instances, you get the following:


MaxBatchSize: Invalid option value: '0' (Namespace: 'aws:autoscaling:updatepolicy:rollingupdate', OptionName: 'MaxBatchSize'): Value is less than minimum allowed value: 1

A few weeks ago, I was asked to scale an environment down to determine if there was any adverse effect in terminating it.  Basically, we would scale down to zero and see if anyone missed it being alive.  In order to do this, we had to go around Elastic Beanstalk and using the Auto Scaling Group:

  1. In the AWS Console, go to Compute -> EC2
  2. In EC2 Dashboard, go to AUTO SCALING -> Auto Scaling Groups
  3. Use the Filter text box to find the auto scaling group for my environment.  Typing in the environment name (e.g. stag-rails-app-s1) should work.
  4. Click on your auto scaling group.
  5. In the Details tag, click the Edit button.
  6. Set the Desired and Min to 0.
  7. Click the Save button.

Deploying Docker Hub Image to Elastic Beanstalk: i/o timeout

By richardtylee.

A few months back we saw the following error while trying to deploy a Docker image to AWS Elastic Beanstalk:


Command failed on instance. Return code: 1 Output: 
[CMD-AppDeploy/AppDeployStage0/AppDeployPreHook/03build.sh] command failed 
with error code 1: /opt/elasticbeanstalk/hooks/appdeploy/pre/03build.sh P
ulling repository username/repo 2015/10/23 22:50:35 Get 
https://registry-1.docker.io/v1/repositories/username/repo/tags: read tcp 10.0.0.1:443: 
i/o timeout Failed to pull Docker image username/repo:latest: Pulling repository 
username/repo 2015/10/23 22:50:35 Get 
https://registry-1.docker.io/v1/repositories/username/repo/tags: read tcp 10.0.0.1:443: 
i/o timeout. Check snapshot logs for details..

At first, it happened intermittently; maybe a third of the time.  After time, it occurred more than two-thirds of the time.  This ended up bottlenecking our QA team.

How to Make Docker Images Smaller: Introduction

By richardtylee.

Docker has become very popular in the last few years.  At work, we've been using it for over a year now.  As we develop, images for our apps are becoming very large. This is becoming an issue because building, pulling and pushing these images are taking longer and longer. Furthermore, our deployment process has been intermittently failing when doing docker push to Docker Hub; this happens particularly often when upstream internet is slow. Our team tried a handful of different techniques to shrink our image sizes.

This week, I'll go over the different techniques to shrink Docker images. We'll look at the advantages and disadvantages.  Depending on your engineering organization and architecture, some will worker better than others.

  1. Base OS Images
  2. Ruby and Rails Images
  3. App Code Separation
  4. Docker Squash