James' World

Nice article on what’s new in DNS in the past 2 years – not much apparently, according to Cricket Liu and ThousandEyes.

Also check out the new DNS book from Mark Jeftovic, the founder and CEO of EasyDNS.com.

dnsdist
O2 ‘to seek millions’ in damages over data outage [due to Ericsson SSL Certificate Expiry in 11 countries] HN

When deploying a production change, usually you have a rollback procedure – documented even! 🙂

But sometimes after a deploy, things don’t work exactly as expected. At that time, you need to decide which is better: rollback, or fail forward?

If it’s a small, isolated code change and you can operate with the old version, often it’s an easy decision to rollback.

But in more complex situations, sometimes it’s better to accept that the change wasn’t perfect, but:

• is still an improvement overall, and gets you closer to your goal
• or is equivalently bad to the previous situation, but in the right direction
• moves to a new target architecture that can’t be simulated in qa or stage for reasons of time, cost or complexity.
• serves as a commonly-understood stake in the ground, or anchor point. “Now that we’re here, we can see the right direction!”

and keep the change and fail forward instead of doing a rollback.

Generally to know if failing forward is an option, you need:

• enough personal and organizational responsibility to accept the risks and handle the consequences.
• a clear understanding of the overall IT systems and IT risks
• a clear understanding of the overall business systems and business risks
• availability of staff to do verification and fix small issues that arise
• to pick a good time for the change that minimizes stress and risk
• Monitoring and application logging tools help evaluate the situation. (I’d even suggest rounding out your tools inventory beforehand if failing forward is new to you.)
• communicate that you may fail forward if necessary, based on calculated, not reckless, risk assessment and that rollback is still an option.

Some actual examples of when I have failed forward successfully:

• firewall rule changes that were closer to the final goal, but broke a couple of servers temporarily.
• database schema changes that were correct, but required a day or two of minor internal application updates that were not in the original QA test plan.

Some actual examples of when fail forward was not acceptable, and rollback was required:

• changes from httpd 2.0 to 2.4 that actually required significant re-QA and updates to the deploy process
• database schema changes that were correct, but required a major application re-build and re-QA totalling more than 3 hours of downtime
• changes that affected legacy applications with no budget for developers or QA.

Especially with databases, the arrow of time cannot be reversed. So database restores results in the loss of data on busy systems, making fail forward the default policy at many SaaS companies. Additionally, failing forward helps with development velocity.

I recently created a new minimal build system, called pico-build, that is hosted on github.

pico-build is the world’s smallest, yet featureful, three-environment (dev, stage and prod) build and deploy system.

It uses make, but in a different way than normal – pico-build manages entire environments, not individual files.

$ make
usage: make [help|check|dev|stage|prod|dist|all]

It is intended for individual programmers and small teams who want to test and deploy to three environments, but don’t want to spend time setting up (and patching) Jenkins or building a CI/CD pipeline.

Folder structure of a website deployed with pico-build, showing the deploy flow in red

To deploy according to the diagram above:

$ make dev
$ make stage
$ make prod

or

$ make all

Please try it out and send me feedback or github pull requests! 🙂

When I was working in Toronto in the mid-90’s, I was offered a game developer role at a downtown company.

At the time, I was one of the top arcade players on Centipede, Millipede, Battlezone and G-LOC. I had also written clones of Centipede for the PC in C and assembler.

They offered me exactly half the salary of another offer, and expected developers to sit shoulder-to-shoulder around wooden picnic tables. They oozed sketchiness, and I already knew how to write a game, so I passed. But hey, they had a blue screen room! 🙂

I moved into Internet Operations, ending up at Netscape. Eventually I did some consulting for a slot machine product which needed to pass Las Vegas gaming compliance. Small world! 🙂

To pass, one of the requirements was that the game had to be guaranteed to start in a known state each time, presumably in case employees or bettors tried to improve their odds by yanking the power and restarting the game.

That guarantee was accomplished by using a journalling filesystem and a built-in UPS with linux services carefully configured to start and stop reliably.

Some links showing how tough the game industry is in 2018:

The tragic end of Telltale Games
infinitroid.com: Did I just waste 3 years?
playstationlifestyle.net: More Than 1,000 Jobs Were Lost in the Games Industry Over the Past Year, 10 Studios Closed Doors