Pelican Publisher Script
When kartones.net was a blogging community and not my current personal minimalistic landing page, one of the blogs that my friend Lobo666 and I maintained was Uncut. With the change to BlogEngine.Net it kept working easily, with a combination of a WYSIWYG editor (or Windows Live Writer) and uploading post images via FTP (minor manual step). But when I recently moved everything to static sites, as Pelican not only doesn't provides any editor but forces you to build the site to preview the changes, my friend was quite impeded to keep posting at the blog.
On the other hand, I already had some post-processing scripts, to cleanup some files that were always copied to the output folder (and thus, uploaded to the site) and to do other tiny tasks like duplicating files (I want to maintain backwards compatibility with the original RSS feed addresses of the old blogs). They were ad-hoc, but after showing them to my friend he just asked me "if I could just make those scripts also upload automatically the modified files". And indeed, making some changes to pass by command line optionally some post identifier (I decided to use the slug) would help. As would too ease things just removing all the "full indexed pages" that Pelican builds (
index<zero-to-almost-infinite>.html pages), and just leaving 10 pages and a link to the full archives page:
This way, and removing the tags, categories and authors subfolders as I don't use them, the number of modified files to upload on a mere new blog post action is around a dozen, making it blazing fast to "deploy" with some Python code. In the end, generalizing the script for the three blogs that I still write and/or maintain, by specifying a few configuration parameters you can specify folders to create or delete, files to copy, remove, duplicate, truncating the index files... and of course upload a post or just build without uploading.
I don't want to extend myself much more as the utility of this tool is limited and very specific, getting to the point, I uploaded the script files to my Python assorted GitHub repo. The direct url of the publisher files is: https://github.com/Kartones/PythonAssorted/tree/master/pelican/publisher.
Usage is quite simple:
python3 publisher.py your-great-post-slug
And to only build:
And that's all. Until next time :)
Recommended Articles - 2017/04/01
As I recently switched job and took a few days of vacations in between, not much relevant to write about on the personal side, so another bunch of relevant articles I've read recently.
- Why Gitlab is not leaving the cloud: Interesting summary of feedback received, seems that predominant point is hidden costs and maintenance requirements that you have with a datacenter and not with the cloud.
- Another chapter in Uber's evil march to hell: This time a tool to evade authorities. Good summary of incidents at Wired
- The Uber Bombshell About to Drop: And continuing with Uber (I'll stop after this because I dislike them but don't wish to spend "bad energies"), good and detailed timeline of the Otto now apparently fake subcompany scandal.
- Wikileaks publishes CIA Hacking Tools: Scary stuff inside, like lots of unknown zero day exploits, malware for remote control and "covert microphones"...
- Firefox 52 released: I mention it because is the latest release with support for Windows XP and Windows Vista, and old plugins (using NPAPI). In exchange, is the first version to come with WebAssembly.
- @_ericelliott: How to speed up developers:
- Give them one task at a time to focus on
- Avoid context switches
- Cut meetings
- Avoid interruptions
- Helpful(?) coding tips from the CIA’s school of hacks: Some obvious, others interesting. And mostly it showcases how their it folks are just humans like everyone else.
- Microsoft is putting OneDrive ads in Windows 10’s File Explorer: It's their operating system but... this is ugly and paves a road to darker destinations.
- Living without expectations: Good advice, but hard to accomplish, at least in the short term.
- Seneca on The Shortness of Time: Of course, with the mandatory mention of not wasting time on useless meetings, but in general advice of making your time count.
- A Software Developer’s Guide to Dealing With Coworkers: Really good advices inside.
- Google, Facebook, Twitter must comply with EU consumer law—or face fines: Good but slow advancements. It is ironic that justice now acknowledges "the growing importance of online social networks" When since around 10 years they boomed and this unilateral power and privacy violations have always happened.
- A Career Retrospective—10 years working in tech: Equal parts sad and great, a tale about sexism and harassment, but also about following your dreams and being creative.
- Forget Feature Requests: Small article but quikcly summarized as:
- Let your customers remind you what's important
- How do you manage them [requests]? You don't. Just read them and then throw them away
- What happened to tablet sales?: Interesting because around me I felt exactly the same that article points out: once you have a tablet, you stick with it as long as possible, like with a computer and unlike with an smartphone.
- Should I stay or should I go? How to decide when it’s time to switch jobs: Great article summarizing main reasons. Identified myself in some of them.
- For sale (in the US): Your private browsing history: USA, "the nation of freedom" (although we could discuss it), but definetly not "the nation of privacy". Another step in losing our rights to our "data" and habits.
- @rbranson: [...] Production software tends to be ugly because production is ugly. The ugliness outpaces our ability to abstract it.
- @codemanship: Don't explain code quality to execs. Explain high cost of change. Explain slowing down of innovation. Explain longer cycle times
- @KentBeck: First you learn the value of abstraction, then you learn the cost of abstraction, then you are ready to engineer
- lobster_johnson: Interesting advice from a company having used microservices for 6+ years: Different services yes, but with a centralized data store to avoid synchronization issues (of local "data silos").
Recommended Articles - 2017/03/05
Had a bunch of links pending but past weeks have been quite busy. It's so sad that unethical and directly wrong company behaviours have been dominating the news ecosystem lately...
- Understanding the GitHub Flow: A quite simple and intuitive way of working with branches with GitHub (but applicable everywhere).
- Trunk Based Development: And the opposite to the previous point, how to work directly with trunk, not using branches. While it requires training, more care and probably some experience, I've been working in the previous step (only
dev branch and then merge to
master) and it speeds up a lot the flow, especially if you do TDD or pair programming.
- Adding Community & Safety checks to new features: Really interesting post from GitHub about non-functional requirements related to community and user safety. Must read as we usually don't take this points into account.
- Trump may sign executive order re-vamping USA’s foreign worker visas: Going to work at USA is getting uglier and harder... If this order passes, H-1B visas could cost way more and be harder to obtain.
- GitLab.com Database Incident + Postmortem of database outage of January 31: Transparency first. A GitLab engineer made a mistake and production data was lost (and in the end couldn't recover around 6h of gitlab.com data), but the exercise of a public incident report, streaming of the ongoing fix and sincere communication is what I really like. After all, we're human and everybody makes mistakes.
- Announcing GVFS (Git Virtual File System): Microsoft is changing so much that is now even working on a special Git version that allows them to work with huge repos (million files and hundreds of GigaBytes). Interesting although only seems to work with the latest Windows 10 builds.
- The Dark Standup: Good example of why forcing working your 40 weekly hours and not more makes you more efficient.
- Getting out of the startup rat race: Couldn't agree more with the article, but it's such a common scenario...
- Report: Pokémon Go has now crossed $1 billion in revenue: It's curious that media was quick to forget it and label it a failure when the growth decelerated, but the income numbers are still great.
- A future without browsers, February 2017: Nice slides about how in a few years the concept of web browser will probably dissapear. Also a quick but nice recap of how we came to the present regarding internet browsing.
- How to be an effective early stage employee. Hint: be helpful: Hint #2: Try to follow the advice whenever early stage employee or "late" one ;)
- The Power of Big Data and Psychographics: "4,000-5,000: collected data points per adult in the US". 12 minutes long talk about big data applied to US elections a few weeks before they ended. Interesting and scary once more.
- Reflecting on one very, very strange year at Uber: Sad story, good advice from DHH (hint: delete your uber account then delete the app).
- Culture is the Behavior You Reward and Punish: Another article about company culture, what you say it is versus what actions actually define it as.
- The 100% correct way to validate email addresses: Great way of explaining that sometimes taking the simple path saves you effort and works almost as good as the perfect path.
- Exponential growth devours and corrupts: DHH is on fire these days attacking the unicorn world of startups! Long essay of how corrupted the ecosystem of startups becomes when all that matters is money made and the final goal is to sell the company.
- The mythical 10x programmer: Redis creator opinion on the subject. Quick read but good lessons inside.
- Summary of the Amazon S3 Service Disruption in the Northern Virginia (US-EAST-1) Region: Interesting because, as happened recently with Gitlab's Database deletion, a small human mistake caused a big problem. We all make mistakes, but this events demonstrate how in an age of automation and complex tooling, we have to be as careful as ever doing any dangerous operation.
- The Story of Firefox OS: Long but worthwile tale of this great idea that didn't worked well.
Hacking Flash Games Example: Clicker Heroes
Note: I reported this technique to the company behind the game back at august 2015. Never got a reply.
I like videogames, so when I read about some "clickers" genre, I wanted to check how they played. Setting aside the mechanics themselves and my personal feelings regarding this kind of games, one title that caught my attention for being better than the average was Clicker Heroes. After playing a while it looked to me as if the difficulty curve was quite exponential, requiring either lots of patience or spending money at in-app purchases, so I wanted to confirm my suspicions.
Checking the game binaries I saw that there was a
HeroClicker.swf, so it was a Flash game. I've already peeked inside and even dissasembled SWFs with Sothink SWF Decompiler, so was my chosen tool.
I started peeking at the insides, and by mere luck I ended peeking the
ImportScreen class. It had a constant called
SALT just below the variable
_userData, so it caught my eye. I ran the game and saw that in the options you can export your data to an apparently encrypted TXT file, and then import it back... hacking my data was way more appealing, and by chance I had a possible attack vector with the import logic.
There was another constant with a maybe too descriptive name,
TEXT_SPLITTER, and scrolling down I found
fromAntiCheatFormat methods, performing MD5 hashes with the salt of contents retrieved from
The sprinkling or scattering algorithm was not hard to read:
- Get user data json and base64 encode it
- Prepare a new array twice the original size of the base64 encoded string
- Place one by one all base64 encoded characters at even positions
- Randomly put an alphanumeric character at odd positions
And then when writing the content of the "encrypted" data to the file:
- Write the new sprinkled array
- Write the
TEXT_SPLITTER constant as it is
- Generate an MD5 of the original JSON data with the
SALT constant and write it
And of course, the inverse process to import the data.
I built a small tool to apply this algorithm using Ruby, and after saving/exporting my character, it did work and I had access to shiny data like the following:
Writing the inverse was easy, and confirmed me that everything worked fine.
This is how the end of an original encrypted file looks:
And this is how a (valid) file encrypted with my tool/script looks. As you can see instead of randoms I just enter blank spaces at odd positions:
The truth is that even cheating, the game gets to some insane levels that you have to either wait a lot or do level grinding (by restarting via "trascending"), so I got bored too quickly. As often happens, tweaking or hacking a game becomes more fun than playing the game itself.
You can get the tool (needs Ruby) from my GitHub and easily see a simplified version of the algorithm.
Even if the code were didn't had so obvious names, as the text splitter fragment can be easily spotted at the end of the file (Look for
Fe12NAfA3R6z4k0z at both screenshots), just doing a classic saved game deltas diff would have raised awareness (whole content of the file would have changed except for that fragment) and made me search the SWF for that splitter string.
Recommended Articles - 2017/01/29
I wasn't planning to post again so quickly, but I've been reading quite a few interesting things and wanted to share the links before the list gets too big.
- The happy secret to better work: Transcript of this TEDx talk about improving your happiness.
- When Gadget Fixers Turn FBI Informants: Members of Best Buy’s Geek Squad passed incriminating evidence to the law enforcement agency and received payments from agents.
- Saving you bandwidth through machine learning: Interesting both as usage of machine learning and as CPU vs bandwith tradeoff.
- Searching For Half-Life 3: While it is an article about a videogame (never done), I liked it because of the explanations of how radically differently Valve operates as a company, with extreme freedom and creativity.
- Reported "backdoor" in WhatsApp is in fact a feature, defenders say: Feature or not, used or not, the option to read the messages is there.
- Robots Will Devour Jobs More Slowly Than You Think: "it’s tasks that can be automated, not whole positions".
- One Thing: Precisely the technique I use when I get stuck or overflown of important things to do. As I like to say, "when everything is important, nothing really is".
- The Humans Working Behind the AI Curtain: Tasks advertised as AI-driven involve humans behind. "The paradox of automation’s last mile": as AI makes progress, it also results in the rapid creation and destruction of temporary labor markets for new types of humans-in-the-loop tasks.
- Restoring Sanity to the Office: "Silent Thursdays. No talk Thursdays" Me wants this. But so full of good points regarding interruptions, work flow, async vs sync communication... Everyone should just read it.
- Metaphors We Compute By: Really interesting article of we (should) use metaphors, but should also be careful of which ones because of their cognitive implications.
- Beware These Nine Design Elements Your Front-End Developers Hate: This last years I haven't done much frontend (and I still don't know CSS) but I've seen some of them, others I had no idea.
- Amazon Is Becoming the Greatest Threat Google Ever Encountered: After everyone mostly only using web search engines for everything, now a "specific" company becomes the starting point again. A curious evolution.
- Google is abusing its powers: Interesting discussion about how when the reliance on obscure (and many times abused) ranking of webpages by Google now has switched to try to force AMP for "faster content and better mobile" while it could have just penalized slow or non-mobile sites.
- Cartapping: How Feds Have Spied On Connected Cars For 15 Years: "almost real-time audio and location data can be retrieved when cops order vehicle tech providers to hand it over". Scary stuff.
- The Infrastructure Behind Twitter: Scale: Insane numbers and also quite interesting both regarding infrastructure and scalability.
- This is why we have working managers at Basecamp (and why Microsoft and Apple stumbled when they lost their tech CEOs): Alghouth radical, has good points (for me at least) regarding management of a tech company, about not being "driven by sales", and about the importance of having good managers. Short but interesting.
- 4 lessons from robots about being human:
- always question assumptions
- when in doubt, improvise
- when your path is blocked, pivot
- if you want to do something well, there's no substitute for practice, practice, practice
- Trump Knows You Better Than You Know Yourself: Although the title is too provocative, psychometrics/psychographics looks really scary regarding building online profiles/personas, as the article points Trump and Brexit used for.
- Look before you paste from a website to terminal: Scary, just read the (small) article and do the test.
- tutorials.ubuntu.com: A few very specific but varied tutorials, just having in common using Ubuntu as the host operating system you follow them.
- Speed Reading is Bullshit: The whole article has good advices, but I like this sentence: "Reading fast gives you two things that should never mix: surface knowledge and overconfidence"