JSON-schema for REST API tests
I wrote a blog post about using JSON-schema to build tests for REST API endpoints that output json at ticketea's engineering blog, so instead of just re-posting I'll just link it: https://engineering.ticketea.com/using-json-schema-for-rest-api-endpoint-tests/
It's a small yet interesting example, and will probably serve me as a quickstart for any future django project as it contains useful bits like:
- classless django views
- an example of pytest-django and two useful features it provides (the
client fixtures). I'm loving the concept of not needing to write scaffolding code
- a small usage example of json-schema :)
Recommended Articles - 2017/11/10
Still a big list but improving from past ones ;)
- "Before scaling up your methodology, consider scaling down the mess. That often is a better solution" @gasproni
- The /bin/true Command and Copyright: An incredibly stupid example of copyrights, thankfully the solution is easy and faster.
- Exploding Git Repositories: Similar to XML bombs and Zip bombs, now how to OOM your repository server.
- Loot Boxes Are Designed To Exploit Us + Gamers Like Opening Loot Boxes Too Much to Stop Now, Even at the Expense of Balanced Gameplay: In the past we had the physical collectible kiosk cards (about movies, TV series...), now is the digital era, but everything is so much simpler to grab your money. I personally hate this mechanics in videogames so much I either don't play them or just cheat the "money" to get unlimited units (if is a singleplayer game), but on the other hand games like Diablo 3 have been abusing the psychology of (perfectly calculated) random loots since before this loot boxes madness... The only videogame I see balancing this pretty well is Hearthstone (but because there I think devs make you pay for playing arenas/tournaments instead of for the cards themselves).
- NGINX Rate Limiting: Interesting article on how to use Nginx's builtin rate limiting features to control HTTP traffic bursts
- Severe flaw in WPA2 protocol leaves Wi-Fi traffic open to eavesdropping + Key Reinstallation Attacks: Breaking WPA2 by forcing nonce reuse: Very severe Wifi issues... read carefully both links to know the scope and small mitigation steps, but let's hope router manufacturers rush to provide patches.
- Introducing Lifecycle Policies for Amazon EC2 Container Registry: ECR is slowly getting better, now supporting specifying policies to kill old container images.
- Finding Truth in History: Some highlights: "History does not reveal causes; it presents only a blank succession of unexplained events"; "...its complexity makes it difficult for us to learn exactly why things happened the way they did"; "Finding truth in history is about understanding that this truth is not absolute".
- One person’s history of Twitter, from beginning to end: A melancholic story of Twitter from its early times to the hate pool it is now. Plus this sentence from the text: "when leadership doesn’t want something fixed it’s close to impossible to fix it. And when leadership doesn’t see something as a problem, it’s not getting fixed at all"
- Our minds can be hijacked: the tech insiders who fear a smartphone dystopia: It seems I'm not alone, and that there are way more extreme cases than just wanting to keep (some) privacy. Very interesting read, which also touches the "variable rewards" topic (loot boxes thing)
- Activision patent matchmaking system pushing microtransactions: If loot boxes issues weren't enough, now this... unbalancing and tweaking matchmaking rules to maximize benefit from buying virtual items and faking for you a sensation of having done well (by matching where your new shiny object outmatches other players who didn't paid for it). So sad to see videogames evolve to this.
- Documents obtained by Privacy International show that UK intelligence agencies may analyse our Facebook and Twitter accounts: The title says all...
- Why Trello Failed to Build a $1 Billion+ Business: "Why I wish I'd sell a company for 425 million dollars". Not a bad read but I personally disagree with the point of view of making your product bigger, fatter and full of features "to better sell it". IMHO Trello decided to stick to a "everyone gets most features" plan and kudos for that (and for the money of the buyout).
- The pitfalls of A/B testing in social networks: On the complexities of A/B testing, community-based A/B testing and specific dating sites challenges. Interesting reading, and I agree with the "you can't really trust anything from an A/B test in social networks" sentence (except maybe if you're Facebook with so many millions of users).
- Adding Kubernetes support in the Docker platform: Unexpected but interesting move!
- Hey Siri: An On-device DNN-powered Voice Trigger for Apple’s Personal Assistant: Extense and detailed (at least for noobs like me) article on how Siri works, really interesting reading.
- Why Surge Prices Make Us So Mad: What Springsteen, Home Depot and a Nobel Winner Know: Really interesting the "please the fans" approach of not too high ticket prices despite demand. Well, and in general the whole article is worth reading. Again Uber gets some hits for being the bad example...
- Windows now includes gaming cheat detection at the system level: Ugh... I'm stuck with Windows 7 for gaming and as time goes by, reasons pile up to keep doing so. Hopefully as almost nobody uses the Universal Windows Platform won't gain traction but I really dislike this invasive "protections".
- Nielsen says it can now measure Netflix streaming: It all sounds "curious" until you realize that in order to measure what you're watching, they have to be listening, right? And with this smart TVs that have already been proven multiple times to send data to the manufacturers, could now mean you might provide that data unwittingly...
- Polygonal Planet Project, a study in tilesets: Beautiful WebGL mini-planet renderer
- Chrome Dev Summit 2017: All the videos from this Google event.
- Pay with Google and speed through checkout: New payment system which promotes both a quick process and no transaction fees... which is nice to hear but makes me wonder if they either want to enter the market or they plan to monetize your shopping behaviours, etc.
- The scale of tech winners: " the four leading tech companies of the current cycle (outside China), Google, Apple, Facebook and Amazon, or ‘GAFA’, have together over three times the revenue of Microsoft and Intel combined (‘Wintel’, the dominant partnership of the previous cycle), and close to six times that of IBM. They have far more employees, and they invest far more."
- Firefox 58 warns you if sites use Canvas image data: Nice that at least one browser cares about this and removes the need for extensions/addons...
- Google Colaboratory: Basically they've setup a free google-hosted jupyter notebook.
- Amazon Aurora with PostgreSQL Compatibility: Amazon keeps growing their could-provided database services... 3x throughput, 6-way replication and up to 64TB, not bad for huge amounts of data!
- Cloud SQL for PostgreSQL adds high availability and replication: And Google Cloud Platform fights back against AWS by also offering their hosted PG with steroids.
- DjangoCon US 2017 and PyGotham 2017: Talks from this two Python-related events are available.
- new "Activity Recognition" Android permission: that shares with apps whether you're walking, driving, sitting... Another darky step, especially since you cannot disable it!
- uncaptcha: 85% success rate defeating Google Captcha (via the audio mode) is quite decent for automation...
- Big data meets Big Brother as China moves to rate its citizens: I'm speechless...
- How the Frightful Five Put Start-Ups in a Lose-Lose Situation: From Spain the scenario doesn't looks as grim, but at least there's hope with some of the new players that enter the market, but it is true that many huge companies eat smaller ones and then they dissapear (and their products die)
- Mobile @Scale 2017 recap: Quite out of the ordinary scenarios but no less interesting talks
- A list of everything that could go in the
<head> of your document: Very cool resource for html templating.
- Documentary on the 80s and 90s Demoscene: The Art Of The Algorithms: Youtube documentary about the Hungarian demoscene. A bit narrowed on scope but still interesting to watch if you like the topic.
- How Netflix works: the (hugely simplified) complex stuff that happens every time you hit Play: Nice entry-level description of how Netflix (but could be generalized to video streaming) works.
- "What's the difference between AI and ML? It's AI when you're raising money, it's ML when you're trying to hire people" @WAWilsonIV
- The Numbers on Your Memory Card Explained: To handle with the mess of speeds, compatibilities and strange codes on them.
- One Bitcoin Transaction Now Uses as Much Energy as Your House in a Week: It is sad that is not a greener option (and seems to be getting worse as transaction calculations cost grows).
- Waymo's fully self-driving cars are here: Despite being a seemingly low-risk scenario (I can’t imagine yet in a huge city center), level 4 self-driving cars and this video look to me like real science fiction.
- GitHub welcomes all CI tools: Nice list of the most used Continuous Integration tools. It is a bit mixed as some like Jenkins don't offer SaaS while CircleCI or TravisCI do, but based on my experience I highly recommend any of the top 3 ones (top 2 if you don't want to self-host and maintain it).
- PostGIS Performance Profiling: Good intro to what happens when you render map tiles regarding using PostGIS and some optimizations.
Overwork and/or crunch time is a recurring topic in the tech industry. From the videogame industry where it can be normal to work +80 hours per week during months to consultancy or startups, more or less everyone has at least once had some overtime experience. After a long time working in the industry (since 2001, fulltime since 2003), I've had varied examples and situations, so instead of just saying "wooo it is terrible!" why not sharing them as a small recapitulation.
Note: This is solely my experience and I talk about my personal overtime experiences, except when explicitly noted.
My first crunch was at a client working at ilitia. We were developing a Windows Live Agents bot (at 2008, who said bots are new in 2017?) for one of the major political parties and we were approaching a hard deadline: Either we shipped before a certain date, or Microsoft couldn't guarantee the bot would be online in time. So, two days before the deadline my colleage and I decided to setup camp at the client office and not move until done. It took 29 hours and so much coffee my colleage's hands were trembling a bit by next day's morning, but we sent it on time.
At Navteq we had a .NET server + J2ME mobile application for the Spanish market used at some relevant events (like "La noche en blanco", a night where most bookstores open at Madrid). When I joined it had severe performance issues rendering the maps and we had an event soon. I took the work laptop home and during most of the night rewrote the entangled server logic so it would serve hundreds of mobile requests per second with the same server hardware.
At Tuenti we kept an internal joke that if we were paid all the extra hours poured in, we'd all be rich (aclaration: we had bonus, performance evals, salary raises, etc.). In my case I tried to stick to the schedule, but there were two recurrent exceptions: I had so much new things to learn I had to frequenty spend hours and hours weekly reading books, articles or source code to be productive coding the next days. And then, more or less yearly we had some big redesign project that usually sucked up the whole tech and got us working overtime for a few months. Sometimes was just a few extra hours per week, other times way more hardcore, like working 13 days in a row, then one sunday "off", usually 10 to 12 hours at the office.
Other examples from other companies are preparing going to launch the new shiny website, that we had been months building and had a branding campaign signed so we had to stay for 20-something hours at the office until we got all the pieces working (and had to do a few horrible last-minute hacks to ship in time), or being at Manhattan debugging problems and publishing hotfixes on a friday night from a bar, switching laptops as the batteries were draining.
In general I think overtime can be justified when it is a special situation, something extra and rare that justifies the extra effort. Overtime causes damage, whenever short-term (sleep issues, zombie-state for a few days, loss of focus) or long term (I left Tuenti in part because after the 4h redesign crunch I got tired and was affecting my personal life). But what it most frequently ends up causing is burnout. I've also seen great, awesome engineers write buggy and messy code due to extended periods of time of sleeping few hours and working insane amount of hours, as tests are sadly the first thing that tends to get cut when under time pressure.
Because of those situations and others I've ommited, I now control a lot my overtime. It is also for the good of my employer, as if I get too tired my productivity will decrease and I'll have more mistakes.
If I'm allowed flexibility I do provide also flexibility: One day I'm late because of a home emergency? No problem, next day I'll make for it. I'm asked to come at 6:30 AM at the office to support a critical release? Fine, as long as nobody complaints if I also go earlier home to rest. But that's not even overtime :)
On the contrary, I've also refused to do on-call when the company was expecting us to do it unpaid and as a norm (truth be said, in the end an on-call was setup).
I'm proud of some of the times I did overtime because I think it was really worth it, other times still agreed to do it because was good for the company (if not so critical), and very few scenarios there was really no other choice without getting into a violent confrontation so I though best to go ahead and take actions later on. But also learning to say no is a skill you develop (and you must).
Things that I do not consider overtime are extra activities in which I participate willingly, although there can be some peer pressure with time you develop resistance to it and I think I've never been manipulated to do them (in fact I quit from some when they weren't fun anymore). Some real examples:
- At ilitia, we setup a CDi ("Club de Desarrollo ilitio"), once per week staying at the office with some beers to do brainstormings, pet projects and think about possible things to build (at the company) to make some money and separate from consulting services. We did product requirement documents, we built proof of concepts, but the general idea and willingness diluted away after a few months
- At Tuenti, we had quarterly HackMeUps, 36-hour hackathons starting thursday. Some were great on the fun part (have drinks and long talks with colleages until late), some on the tech side (quite a few projects ended up in production after some polishing)... I didn't won any but tried to participate in most and usually stayed either all night or until late
- Despite being shy, I like explaining what I know (not much), usually giving talks at user groups and sometimes conferences. I might spend some work time on thinking and peer-testing the talk, but I prefer to spend a few hours at night without any rush thinking the content, the structure, the main goal...
"Work hard and go home" - Slack company motto
Recommended Articles - 2017/10/14
Again long since I last wrote a list of articles so a few might even look "old news". Plus way way more articles gathered than I'd like to, but also lots of reading opportunities with so many topics :)
- Video game developers confess their hidden tricks at last: Good summary of the lenghty Twitter thread, interesting to see how we get cheated often to feel better than we really are :)
- The Right Way to Manage Secrets with AWS: Self-explaining if you use AWS and know what KMS is.
- NGINX Unit: This can be really interesting, a small application server that promises zero-downtime configuration changes and easier updates.
- Lessons Learned Scaling Airbnb 100X: Product-related advices are nice, encouraging you to not fear crunches, I tend to disagree except if they are really rare.
- Pub-Sub the swiss army knife (tech pill): Another post from my ex-colleage Eferro, this time explaining how pub-sub works and where it fits best.
- New in Chrome 61: I don't usually write about browser new features, but one caught my eye: WebUSB!!! This can be so fun combined with web bluetooth and other pieces like service workers!
- Atlassian launches Stride, its Slack competitor: This chats and bots flood is curious, I wonder if is really worth all this hype. And let's remember that Atlassian already has Hipchat, cannibalizing your own product sounds strange... except if you plan to phase off the old one.
- Titan in depth: Security in plaintext: Google is building its own security chips for their servers. The kind of things you can only think about doing when you have so much money... but still a nice move.
- Azure Confidential Computing will keep data secret, even from Microsoft: And Microsoft is also adding hardware-based security to their cloud platform...
- As Amazon Pushes Forward With Robots, Workers Find New Roles: Not everything is bad news regarding "robotized workers", and also a good inside view of an Amazon warehouse.
- Introducing Atom-IDE: Atom releases IDE extensions because... it wasn't IDE enough already? I really don't get why, except for getting extra news coverage and/or to keep Atom "core" minimal as it is opensource, but hey, whatever eases our developer lives is welcome.
- Here's What Security Experts Think About The iPhone X's New Face ID Feature: "What if a cop stops you and points the phone at your face, one Twitter user asked, while they have you in handcuffs — then he or she proceeds to look at your phone without a warrant?"
- Face ID Raises Some Scary Questions—Here Are Some Answers: "Face ID is supposed to improve on this by requiring 'user attention' Basically, this means you have to have your eyes open and make eye contact with your phone to get it to unlock" And, if you're completely drunk, will it unlock too?
- What you should know about privacy and Apple's FaceID on iOS 11: Another article. Three in total because while cool and handy to use it raises a few triggers on my personal privacy and security alarms.
- Google Chrome will block auto-play video starting January 2018: But actually since Chrome 61 you can already do it, just type as a URL
chrome://flags/#autoplay-policy and select
Document user activation is required.
- Founder of Fan-Made Subtitle Site Convicted for Copyright Infringement: Sad news, it seems you're not even allowed to create your own subtitles "without authorization", at least in Sweden. Another laws nonsense...
- The React license for founders and CTOs: Although the license was finally changed, an interesting reading on why it was initially released as Facebook BSD+Patents.
- "15 years ago, the internet was an escape from the real world. Now, the real world is an escape from the internet" -@Noahpinion
- Devs unknowingly use "malicious" modules snuck into official Python repository: I wouldn't thought that companies should security-audit code from official repos, but it seems is a good idea lately...
- AMP vs non-AMP: Most interesting point that many of us have suffered with integrations of different kinds: "It's important to recognize how much extra work things like AMP, Facebook Instant Articles, Apple News, whatever Amazon dreams up next, etc. etc., dump on your development team -- the maintenance alone can swallow up countless hours"
- Per-Second Billing for EC2 Instances and EBS Volumes: Name says all, finally one of the big critics of AWS gets solved. No more leaving the machines up for 55 minutes to "squeeze them as they are paid for a whole hour".
- World Wide Web Consortium abandons consensus, standardizes DRM with 58.4% support, EFF resigns: DRM extensions have been officially approved by the W3C, and the EEF resigned due to that decision.
- How Booking.com manipulates you + Truth about Booking.com: Yet another story of not so good working environment, but in this case with user manipulation added to the mix. Some really nasty UX and psychological tricks.
- Here's Why You Should Have a CAA DNS Record for Your HTTPS Website: Proposal to add a Certification Authority Authorization (CAA) DNS record so browsers can check if the certificate a webpage is serving is valid because the CAA record says the issuer is valid/allowed.
- New in postgres 10: Lots of improvements, especially partitioning and replication related.
- Google to acquire HTC's Pixel smartphone team for $1.1 billion: Good move, they leave HTC independant but at the same time adquires and moves to Google their best engineers, securing the future of GPhones...
-SRECon EMEA 2017 Notes: Notes, links and additional resources of the conference.
- Low-complexity leader election with AWS: Simple algorithm to perform EC2-based leader election with AWS.
- Collection of free ebooks from Packt Publishing: Self-explanatory names, the books that Packt gives for free during 24h.
- One Tinder user's data request turned into 800 pages of probing info: The subheader is perfect "when a service is free, you are the product".
- The Mega HTML5 Cheatsheet: As long as interesting indeed!
- How I hacked hundreds of companies through their helpdesk: Social engineering and phising worry me much more than software/hardware security, harder to "fix".
- Have Smartphones Destroyed a Generation?: Interesting read, as I'm past mid 30s I no longer know youngsters' habits so well, but it is true habits are changing (and for all ages).
- Just Do As Expected: Like the general guidelines:
- Everyone is responsible
- Continuous Improvement
- Respect others and their opinions
- Bias towards action
- Code as expected
- Code Quality
- Ship it!
- Transitive property
- Android users rejoice! Linux kernel LTS releases are now good for 6 years: Really good news, as the phone is something we use too much to have it become obsolete as quickly as happens now (1/2 years max, depending on when the OEMs add their layers of "stuff").
- 1 Trillion HTTP Requests Per Month: More than the actual throughtput what I like from the article is the evolution story, how they faced different problems and how they scaled and rewrote things.
- Apple is really bad at design: I'm no designer but the examples are really good arguments...
- Apple Collecting Browsing Data in Safari Using Differential Privacy in macOS High Sierra: Ah, if instead of a bitten apple we were talking about a Redmond multicoloured windows logo, this would smell to lawsuits, but being a "cool company"
- Apple Open Source: But for once, not everything is bad, they released all their kernel source codes!
- Free Selenium Tutorials: Really huge, detailed and very well explained compilation of Selenium tutorials, from begginer to advanced topics. Highly recommended if you build browser tests with the tool!
- Open a terminal and type
telnet mapscii.me (more details)
- "Former Twitter employee reveals what we can all observe. Reducing abuse on the platform a non-goal since it hurts 'engagement'"@Carnage4Life: I loved the platform but is slowly declining between the rage, hate, manipulation and not caring about quality of content...
- Tfl plans to make £322m by collecting data from passengers' mobiles via Tube Wi-Fi: And again, when you don't pay with money, you pay with your data...
- Breaking Up the Behemoth: How applications usually evolve, and how to keep complexity of them under control. Nice advices, go read the full post.
- Streams: a new general purpose data structure in Redis: I love redis, and the streams look more and more as a great alternative to Kafka or AWS Kinesis streams. The article is long but enters into a lot of detail about what to expect from streams (spoiler: they are so well thought).
- Google Chrome Will Block Tab-Under Behavior: Didn't knew this dirty trick but nice to see them
- Web and Android Scholarship Program: Google and Udacity are offering free scholarships to learn web (frontend or backend) and Android development.
- Microsoft Edge for iOS and Android: What developers need to know: And why the hell would you do that...
- Windows Phone is now officially dead: A sad tale of what might have been: ... until you read this article and then see that Windows Phone is not anymore de facto dead but officially too.
- Explaining the new cryptocurrency bubble—and why it might not be all bad: I do still think we're playing at a dangerous game here, but it is true without doing things different there would be no change or innovation.
- A 1 KB Docker Container: Demoscene for devops X-)
- How Kaspersky AV reportedly was caught helping Russian hackers steal NSA secrets: Woah, need to read this one slowly, hackers detected by hackers who were inside the system already. If wasn't so convoluted around computer hacking would make for a nice spies movie.
- The Future of HHVM: Facebook's compiler for PHP will stop said compatibility, not only regarding PHP7 but also regarding PHP5. They want to improve their improved version, Hack, so no more looking back.
- Extending per second billing in Google Cloud: Excellent answer to the recently announced AWS per-second instance billing.
- The SQL I Love. Efficient pagination of a table with 100M records: Nice solution and explanation
- The Absurdly Underestimated Dangers of CSV Injection: Amazing that both MS Excel and Google SpreadhSeets can be exploited with injected code. Short read but very relevant if you use any of both tools.
- Silicon Valley: Interesting analysis of how Apple is winning on the hardware side by getting the edge on microprocessors, sensors, etc.
- "Space X sends a rocket up into space. Lands back on its feet back on earth 7minutes later. I can't even run an npm install in that time."@toddmotto
- Pornhub Launches AI-powered Model That Detects Over 10,000 Pornstars in Videos Using Computer Vision: Porn industry always adapting itself to technological advances :)
- I Wrote a Hit Song With Justin Bieber. Want to See My Royalties?: Wow, incredibly low numbers from streaming earnings, something's broken on the pipeline because numbers are really really low...
- High-impact refactors keeping the lights on: Slides of my talk given at the PyConES and a local meetup (one of the reasons this post got delayed).
- Remote Work: Why it's cool to work in underpants: And slides of an internal talk I gave at my previous job about how to improve remote work and become trully "remote first" (insted of just "remote friendly").
Book Review: Game Engine Black Book: Wolfenstein 3D
Title: Game Engine Black Book: Wolfenstein 3D
Author: Fabien Sanglard
I've been following Fabien's blog for many years, with his excellent videogame source code reviews and in general great articles, so when he announced he was finishing a book about Wolfenstein 3D I got so excited and actually preordered it (on Google Play/Android, read from a 9" tablet).
You can actually grasp the contents of the book not only by reading the whole author's blog, but with certain examples, like how the fizzlefade effect was built or how to understand floating point. But to give a general overview, it is a quite specific technical book: You get history about 90s hardware (the 80386, VGA graphic cards, sound systems and the dreaded MS-DOS RAM handling), some background about the team and who did what regarding Wolf3D, but then it is a dive into the source code, analyzing, explaining and commenting the most relevant parts of the game internals, from how the raycaster works to how the sprites were stored, the different subsystems (sound, input, memory, cache, ...). It feels short but is +300 pages long, and also contains appendices and sequels & ports chapters to learn miscellaneous things related with the game.
I loved reading it, mostly because Wolf3D was one of the first PC games that actually made me start enjoying my father's 386DX instead of only my beloved AMIGA 500, and while it wouldn't be until 1994 that I started learning to code and grasped VGA graphics programming and assembly, I love the amount of tiny but interesting details that I've learned, from how a digital joystick worked and would be read its inputs, to why there were different sound formats or how clever were the guys at id storing the sprites to speed up even their reading and writing with VGA planar modes.
A great technical and nostalgic walkthrough the source code of a game that changed everything, but also a reading about how at the early 90s and the old MS-DOS days everything was harder, from playing videogames (with the conventional memory limitations) to developing them (way harder than being able to play them :)
As I was reading a book about generating computer mazes I read the book announcement and couldn't resist to try to adapt the maze generators to export Wolfenstein 3D maps. The result can be seen here, as Wolf3D doesn't supports mods or external maps easily, I decided to reverse-engineer NMap's (an unnofficial mapping tool)
.LEV file format and generate those files with simple data.
After reading the book now I want to do a raycaster (probably using PyGame for drawing :_)