On Elastic Beanstalk, Docker and CircleCI

I joined ticketea's engineering team last month, and apart from learning how things work and doing some bugfixing weeks (to get comfortable with the code and peek at some of the projects), I also got assigned to one of the new projects. There are three projects that we have started from scratch, allowing us to decide if to keep or change the current platform (which could be more automated). In order to take decisions, we did some research and proofs of concept.

The main goal of the research was to setup a basic AWS Elastic Beanstalk orchestation system, to allow us to perform deploys, local runs, etc. without needing to manually handle EC2 instances and build the corresponding toolset, as we don't have any systems team.

Our results are mixed but still subject to change as we haven't yet discarded or decided for a certain route, we keep exploring multiple paths with the projects to decide later. Despite that, I'll leave here some notes and references. Don't expect great notes as this is more of a cleanup of a worklog/checklist (actually, it was a simple Github issue).

Update: I wrote this blog post which might be of interest as shows how to access an EC2 service from a docker container running with Elastic Beanstalk. Update II: Blog no longer accessible but you can download the article in PDF from here: Securely access AWS Parameter Store from your Elastic Beanstalk Docker containers

---

CircleCI

We'll stick with CircleCI as our test runner, builder and probably continuous deployment tool for staging. Version 2.0 works nicely with containers and, despite being heavily modified from v1.0, modifications were quick to perform.

• Some caching resources:
  • https://circleci.com/docs/2.0/configuration-reference/#savecache
  • https://circleci.com/docs/2.0/caching/
  • https://circleci.com/docs/2.0/docker-layer-caching/
• Automatic deploys of dev branch was easy to implement
• If you plant ot use Docker, forget about 1.0, it implements and old version that gives headaches. With 2.0 you can install latest one in the "Virtual Machine" configuration and have no issues.

Elastic Beanstalk

EB has been relegated to staging/production deployment. For that, the cluster features (load balancing, rolling deploys, etcetera) are great and very easy to use. Instead, for local development it is between painful and directly impossible without hacks to work decently. The reasons are multiple, primarly being:

• You cannot use docker-compose as EB internally uses it and forces you to use their YML config files or rely on fully manual Makefiles + raw Docker
• eb local works only on pretty much factory-default scenarios. As soon as you start working on real services, it just doesn't works
• EB works using environments, but it is configured so one "folder" is the equivalent to one environment. So having dev, staging, production etc. means one of the two following hacks:
  • Have a single root dockerrun.aws.json with placeholder variables that you replace by the appropiate enviroment values
  • Have multiple dockerrun.aws.json at subfolders (one per environment) and move them via Makefile or similar to the root depending on where you run it
• We've become more proficient on using "raw" docker, but in the end we decided to still use docker-compose, even if only for development. It saves you a lot of command line writing and is quick to change.

Resources:

• http://www.glynjackson.org/weblog/tutorial-deploying-django-app-aws-elastic-beanstalk-using-docker/
• http://www.glynjackson.org/weblog/django-aws-elastic-beanstalk-docker-2/
• http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/docker-singlecontainer-deploy.html
• http://docs.amazonaws.cn/en_us/elasticbeanstalk/latest/dg/create_deploy_docker.html
• http://cloudacademy.com/blog/how-to-deploy-docker-containers-on-aws-elastic-beanstalk/ <- Nice high level summary of what EBS provides
• https://medium.com/@grudelsud/continuous-integration-with-docker-on-amazon-elastic-beanstalk-44fa89024502 <- Very similar to our desired setup: Circle CI, Docker, Python,... Uses images instead of local Dockerfile
• http://abhipandey.com/2015/09/elastic-beanstalk-deployment-automation/
• https://github.com/0xadada/dockdj
• http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/ebextensions.html (sample: https://s3.amazonaws.com/elasticbeanstalk/extensions/ElastiCache.config)
• Task definitions (for Dockerrun.aws.json): http://docs.aws.amazon.com/AmazonECS/latest/developerguide/task_definitions.html
• https://blog.eq8.eu/article/aws-elasticbeanstalk-hooks.html
• http://stackoverflow.com/questions/39083768/aws-docker-deployment <- to fix permission issues
• https://medium.com/learnings-in-and-around-sharetribe/using-aws-ec2-container-registry-to-host-docker-images-for-deployment-with-elastic-beanstalk-b5f21c3c8e21 <- deprecated

EB Configuration files

• https://medium.com/trisfera/getting-to-know-and-love-aws-elastic-beanstalk-configuration-files-ebextensions-9a4502a26e3c
• http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/ebcli-compose.html

Alternatives to EB

One of the teams, after asking for some feedback to friends and colleages is testing Terraform. It looks promising and is working fine for them but also needs maintenance, so there is no firm decision yet regarding if to use it or stick to Elastic Beanstalk and Makefiles (at least for now).

ECS + ECR

We setup a registry and pushed both development and production images after successful builds. It works quite nicely and the only reason we are not using them actively is to try to avoid the permissions hell you enter once you want to share images between different Amazon accounts (not just IAM users on the same account, but fully separate ones).

• https://circleci.com/docs/1.0/continuous-deployment-with-aws-ec2-container-service/ + https://github.com/circleci/go-ecs-ecr <- What we wanted
• https://blog.codeship.com/aws-registry/: Not exactly our CI system, but generally interesting regarding how to setup credentials, what to do at CI system, etc.
• CircleCI + AWS ECR/ECS: Indicates which env vars to setup for AWS credentials
• ECR CLI <- discarded/not needed

Redis

We are using Redis for our project, a docker image for development and Elasticache for staging and production.

• http://stackoverflow.com/questions/26528395/how-to-install-and-configure-redis-on-elasticbeanstalk
• http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/customize-environment-resources-elasticache.html

Tools/extensions to check and add if interesting

• https://joeferner.github.io/redis-commander/ + https://hub.docker.com/r/tenstartups/redis-commander/ <- Didn't end up testing it as current usage of Redis is quite simple, might be of use in the future
• dockercfg file format