Having a good, disposable devbox

Around 2005 virtualization already was working nicely, and at work, as we did .NET consulting, we started using virtual machines (with Virtual PC) as our main development environment. We would have a base VM snapshot with a Visual Studio, and then when starting a project we'd just clone it and add specific requirements (e.g. SQL Server). It was pretty much manual but still a great improvement over having to clean or even format your host machine between projects. Also, migrating to new hardware was seamless, just copy the VM image and good to go.

Now, things have evolved a lot, and having switched to a mostly opensource stack and Linux development environment there's a wider array of options, plus greater automation capabilities. I've never been directly involved in the management of development tools resources or projects (except one or two small scripts) until recently, but I've tried to use whatever environments they provided, with varied results.

Keeping the "optimal" scenario of having everything on your machine (is the fastest and quickest in the short term but has lots of disadvantages too), I moved from manually managed local VMs to having remote dev machines, where you would rsync files, SSH when needed to restart a process, and usually deployed your code to another location (being web dev, mostly having a local dev with your kartones.localhost.lan and remote webserver like kartones.xxx.dev). This approach is not bad (as worked for me for quite a while and in multple jobs) but has two big disadvantages:

• No connection means can't work: You have to setup one (or more) backup DSL lines with a different ISP at the office for outages (which at least in Spain are not so infrequent)
• The noisy neighbor problem: If you share your machine with another 4 developers and your build process is CPU or IO heavy, or you must run some Hadoop map-reduces, you can easily eat all resources and impede other's work

Being few people the remote dev machines is a good approach, but as you grow it becomes a severe limitation.

So, how do we solve this issues? Well, thanks to Virtual Box, Vagrant and Puppet we can now easily have provisioned development virtual machines: Local but instrumentalized VMs that closely match a production server and whose configuration and installed packages are managed from the same tool that setups production machines, just requiring different config sections (but mostly being a copy + paste + rename task). I've lived three iterations of this approach at different jobs, from a quite manually (and badly working) version, to a "working but not smooth enough to replace a local dev env" and to my current job setup, which works so nice we now don't support anything excepting the devbox.

It took us weeks of iterating and forcing the whole tech team to install it by themselves, just following the README instructions and providing either feedback or directly commits with improvements, but feels worth it because:

• We're all on the same enviroment: Problems "are shared" so quickly solved and easily reproducible. If something breaks, breaks for everyone so no broken windows effect
• Process is dead easy to follow: I try to push every service and tool to have a README.md detailing instructions, but in this case the more we use it the easier it gets and more we improve and automate it
• Fast and isolated: Not native-speed but the faster your hardware the faster it goes, and you never hurt other team members' speed with heavy scripts you run
• No need to depend on external storage for a backup: In the past, I used to carry a USB Donge with a backup of the VM just in case the original died, had a fatal update (Ubuntu is great but I've more than once had some update break the VM at boot) or you just want to revert some undesired change
• Almost the same environment as production: This ultimately depends on you, but the closer you get to replicating production the easier you triage configuration issues regarding web server, caching, connection pools...
• Easily updateable: Linux kernel updates, provisioned software updates, individual repository dependency updates... Everything handled via single commands
• Everybody participates: Have an idea to automate something? Code it and push it!
• Helps keeping codebases homogeneous: Having templates for microservices and web-apps is handy, but having lots of services that have to be configured, launched, tested, etc. means you naturally try to setup conventions of folder and code struture, helper scripts, launchers/runners... Make easy doing things the right way and it will yield better results (or at least make hard to go wrong!).

We bet so hard on having this process quick, easy and painless that if I was allowed to, I'd setup the devboxes to self-destruct after 2 weeks of use, to force everybody to re-install them and be always sure that no matter what happens, you can reprovision and have a working dev environment in a few minutes. I manually do delete mine (including the code repositories) and you feel at peace and calm when you just do:

1. Clone the operations repository
2. vagrant provision
3. vagrant ssh + run install.sh script providing your desired username

And this is just the beginning, now with containers with Docker & the like we're moving towards an "optimized" version where you can replicate something really like production, in your local machine, with disposable instances, always updated (and using the same mechanisms than production, to avoid nasty errors) and doing a much better resource usage. But I have not talked about them because we haven't yet migrated to containers, so I have much to learn and experiment before being in a position to give an opinion, I'm just eager to try it!