Kartones Blog

Be the change you wanna see in this world

Book Review: Speccy Nation

Speccy Nation

Last month I forgot to post anything so here comes one of the books I read some weeks ago, Speccy Nation. Small, cheap and not especially good, but hey, I paid less than 2€ for it so what could I expect... Anyway, the full summary at the Book Reviews section.


Migrating from BlogEngine.NET to Pelican

Last weeked I found some spam at two of my BlogEngine.NET blogs. It is not the first time, and in the past updating to the latest major solved the issue, but this time I had to switch from 2.9.X to 3.2, and I already suffered a migration from 1.9 to 2.0 that gave quite some headaches. One of my main reasons to go far away from Wordpress was to stop this tiring battle between spammers and new versions, that forced you to update way too frequently or face serious security bugs and spam-holes. Combine that with a general feeling of being tired of big, admin-driven blog engines, and I needed a big change.

My premises were:

  • Administration fully optional: even prefered not to have one
  • Static site generation: No more security issues, plus FTPing the changes once or twice per month is more than enough considering my posting frequency
  • As less setup requirements as possible
  • Local testing: Including (for the near future) Windows support without extra effort
  • Favor Python 3 over other languages: I want to improve my Python n00b skills and the best way is to use it as much as possible

Reading some articles and checking some static generators, I found Pelican. I peeked at bit at the source code, documentation and plugins, and it looked quite simple. Did some local tests, wrote some posts using existing content and read the documentation, and decided to keep it.

The setup is so easy I got a blog running locally with some base theme in a few minutes. But I had one issue, all my posts are in XMLs (thankfully I had chosen not to use a DB for storage) and Pelican uses Markdown... so I had to transform the data somehow.

I don't do too complex stuff like <meta> keywords and I set some post tags but not even show them, so in general I just needed a few fields for the pelican post format:

Title: ...
Slug: ...
Date: ...
Tags: ...,...

...CONTENT (which can be directly HTML)...

The XML has a very simple structure, with all the fields and just HTML-encoded the content, some regular expression searches and reverse transformations were enough to port everything. I have plans to migrate another blog, and prefer to be able to reproduce the whole migration any number of times, I built a small script. It only handles basic stuff and doesn't extracts other "basic" fields like authors, and only works with posts (I manually migrated the pages as I had few), but it does what it does well and might be of some use to somebody else.

You can find both my scripts and a small plugin (to limit RSS/Atom syndication feed output to only a certain amount of items, as by default dumps every post ever written) at my GitHub.

I'll probably work on more small plugins in the future, as I have some improvement ideas regarding output content generated, but I can't be happier with the results. A proper example of "the Python way of life", simple yet practical code, easy to setup and use and does it's job without many features.

UPDATE: Added another small script to GitHub to do some post-generation tasks like creating duplicates (as "aliases") and moving or removing certain files from the output folder.


Book Review: Building Microservices

Building Microservices book cover

Due to my new job I wanted to learn about microservices, and this book had good reputation and recommendations, so I decided to read it. After finishing it yesterday, I can only recommend it to anyone starting with this system design principles. For a detailed review, head over my books review section.


KISS

"Keep It Simple, Stupid": A nice design principle, usualy forgotten in the development world.

Over the last years I've switched from the (flawed) mentality that whatever I should do must be perfect from the start to being more pragmatic and getting something build fast to then iterate. In the past, I tried doing all kind of mega-extensible pet projects, TDD-driven map generators, complex videogame attempts... and most of them lie dead in states between "not even working" and "proof of concept". Instead, I've built more "games" using only Javascript (hint: 2 dumb ones) and not caring about perfectionism. In fact, I have built quite a few things with almost or no tests, without complex design patterns or fancy language features, and aiming to have a working version in a few hours to then improve it.

This post is mainly a list of those small pet projects and tools, nothing to be proud of technically nor visually, but that really do help me optimize my time and/or ease some tasks. So, here comes the list:

  • A CSS-less web tools toolkit, using only Javascript
  • A bookmarks manager, first attempted using Javascript, later finding that mere server-side C# Arrays and StringBuilders did the job way faster... plus literally 15 lines of javascript to handle contract/expand groups
    Bookmarks web-app
  • A movies & videogames catalog, using only Javascript and the fantastic CartoDB CSV import and javascript SQL API capabilities . Just a search input and options to search games or movies
  • An internal Google-Analytics like poor-man dashboard, again using only Javascript and CartoDB's SQL API. This one has no security so I haven't even uploaded it, running instead locally (if you don't need it online, don't do it)
    Visits Dashboard web-app
  • A minimalistic Python 3 static site generator (including minimization of HTML) that outputs HTML, which I use to maintain my small portfolio site
  • A shopping lists mobile web-app, so that my girlfriend and I can "sync" what needs to be bought without writing and managing paper post-its. Pure C# in ASP.NET web forms with no code-behind and storage based on text files: files are the lists, lines inside the products, last space acts as delimiter of item state. Thanks to Bootstrap and jQuery works perfectly in mobile without any specific line
    Shopping list web-app
  • Small photos uploader, using PHP. Just some file upload inputs and items listing
  • Social media privacy housekeeping tools, using Ruby. Old tweets eraser, uploaded photos remover and other internal scripts
  • An instagram tag-based searcher that builds a crappy HTML with the results so I can quickly browse from my PC photos by tag
  • A Windows oldie but goldie batch file that performs backups (using 7zip) of local relevant folders and leaves the compressed files on an external drive. Hint: if compression is not worth the time, just store files (zip without compression), moving a few big files is way faster than many small ones.

And probably more small tools that I now don't remember, like snippets to generate base64 URLs for to embed images into HTML, hash calculators,... My goal as a developer is to make my life easier, because work already provides the challenges and hard thinking.

All of those took me between hours and less than two nights of coding to build an initial fully working version, and I sometimes evolve or improve little things, but from day 1 or day 2 I have something working. They usually are ugly as hell (and I overabuse Bootstrap) and some use jQuery while others have vanilla Javascript. Also, each is done in a different language, maybe because I wanted to toy with it, or maybe I was using it at work and wanted to train more, or I just wanted to change the language I code into often. The tool is not as important as the results.

One thing all this examples don't imply is the need to build everything from scratch, that would be dumb, it just means that for certain scenarios, you don't need to strive for perfection. I for example use a blogging platform not made from scratch but existing, but I simplified it a working but really old engine to the simpler, not DB-based BlogEngine.net it currently runs with. It has some tweaks and optimizations but maintenance is easy and a local copy for development required 3 minutes of configuring Internet Information Server at Windows.

In the end, what matters is to have something working and adding value, instead of a fancy 90% code coverage application that doesn't even have a single feature fully implemented. My "home code" won't win any award but it works great.


Having a good, disposable devbox

Around 2005 virtualization already was working nicely, and at work, as we did .NET consulting, we started using virtual machines (with Virtual PC) as our main development environment. We would have a base VM snapshot with a Visual Studio, and then when starting a project we'd just clone it and add specific requirements (e.g. SQL Server). It was pretty much manual but still a great improvement over having to clean or even format your host machine between projects. Also, migrating to new hardware was seamless, just copy the VM image and good to go.

Now, things have evolved a lot, and having switched to a mostly opensource stack and Linux development environment there's a wider array of options, plus greater automation capabilities. I've never been directly involved in the management of development tools resources or projects (except one or two small scripts) until recently, but I've tried to use whatever environments they provided, with varied results.

Keeping the "optimal" scenario of having everything on your machine (is the fastest and quickest in the short term but has lots of disadvantages too), I moved from manually managed local VMs to having remote dev machines, where you would rsync files, SSH when needed to restart a process, and usually deployed your code to another location (being web dev, mostly having a local dev with your kartones.localhost.lan and remote webserver like kartones.xxx.dev). This approach is not bad (as worked for me for quite a while and in multple jobs) but has two big disadvantages:

  • No connection means can't work: You have to setup one (or more) backup DSL lines with a different ISP at the office for outages (which at least in Spain are not so infrequent)
  • The noisy neighbor problem: If you share your machine with another 4 developers and your build process is CPU or IO heavy, or you must run some Hadoop map-reduces, you can easily eat all resources and impede other's work

Being few people the remote dev machines is a good approach, but as you grow it becomes a severe limitation.

So, how do we solve this issues? Well, thanks to Virtual Box, Vagrant and Puppet we can now easily have provisioned development virtual machines: Local but instrumentalized VMs that closely match a production server and whose configuration and installed packages are managed from the same tool that setups production machines, just requiring different config sections (but mostly being a copy + paste + rename task). I've lived three iterations of this approach at different jobs, from a quite manually (and badly working) version, to a "working but not smooth enough to replace a local dev env" and to my current job setup, which works so nice we now don't support anything excepting the devbox.

It took us weeks of iterating and forcing the whole tech team to install it by themselves, just following the README instructions and providing either feedback or directly commits with improvements, but feels worth it because:

  • We're all on the same enviroment: Problems "are shared" so quickly solved and easily reproducible. If something breaks, breaks for everyone so no broken windows effect
  • Process is dead easy to follow: I try to push every service and tool to have a README.md detailing instructions, but in this case the more we use it the easier it gets and more we improve and automate it
  • Fast and isolated: Not native-speed but the faster your hardware the faster it goes, and you never hurt other team members' speed with heavy scripts you run
  • No need to depend on external storage for a backup: In the past, I used to carry a USB Donge with a backup of the VM just in case the original died, had a fatal update (Ubuntu is great but I've more than once had some update break the VM at boot) or you just want to revert some undesired change
  • Almost the same environment as production: This ultimately depends on you, but the closer you get to replicating production the easier you triage configuration issues regarding web server, caching, connection pools...
  • Easily updateable: Linux kernel updates, provisioned software updates, individual repository dependency updates... Everything handled via single commands
  • Everybody participates: Have an idea to automate something? Code it and push it!
  • Helps keeping codebases homogeneous: Having templates for microservices and web-apps is handy, but having lots of services that have to be configured, launched, tested, etc. means you naturally try to setup conventions of folder and code struture, helper scripts, launchers/runners... Make easy doing things the right way and it will yield better results (or at least make hard to go wrong!).

We bet so hard on having this process quick, easy and painless that if I was allowed to, I'd setup the devboxes to self-destruct after 2 weeks of use, to force everybody to re-install them and be always sure that no matter what happens, you can reprovision and have a working dev environment in a few minutes. I manually do delete mine (including the code repositories) and you feel at peace and calm when you just do:

  1. Clone the operations repository
  2. vagrant provision
  3. vagrant ssh + run install.sh script providing your desired username

And this is just the beginning, now with containers with Docker & the like we're moving towards an "optimized" version where you can replicate something really like production, in your local machine, with disposable instances, always updated (and using the same mechanisms than production, to avoid nasty errors) and doing a much better resource usage. But I have not talked about them because we haven't yet migrated to containers, so I have much to learn and experiment before being in a position to give an opinion, I'm just eager to try it!


Book Review: Code - The Hidden Language of Computer Hardware and Software

Code: The Hidden Language of Computer Hardware and Software book cover

As now I travel to work via underground, I have more time to read. The first book I've fully read is Code, from Charles Petzold, and I really encourage anybody wanting to learn how computers evolved up to late 90s to go grab it. As for the full review, as usual at the book reviews page.


Page 1 / 78