I’ve become a big fan of Ryan Holiday’s work. It comes as no surprise that one of his books, Ego Is The Enemy, has inspired me to write Every Day We Must Sweep a couple of weeks ago. I couldn’t help it and also read the critically acclaimed predecessor, The Obstacle Is The Way, shortly after. It too turned out to be a remarkable book, drawing on the vast experience of Emperor Marcus Aurelius and other famous historical figures. Again, I highlighted dozens and dozens of passages I considered to be noteworthy. One particular chapter stood out to me, though. It’s called “Anticipation (Thinking Negatively)” and it has a direct, almost disturbing relevance to my daily work. If it were possible, I would give you a copy of it right now, but I’m afraid my interpretation and lots of quotes will have to do for the time being.
The chapter begins with a discussion of premortems, which are said to be popular in the business world. The basic idea of a premortem is that managers encourage employees to think negatively – in terms of worst-case scenarios – when preparing for major events like a product launch. Holiday writes:
In [a premortem], we look to envision what could go wrong, what will go wrong, in advance, before we start. Far too many ambitious undertakings fail for preventable reasons. Far too many people don’t have a backup plan because they refuse to consider that something might not go exactly as they wish.
Astute readers will notice the play on words here. Premortem is, of course, the opposite of postmortem. With a postmortem, we’re examining something after it happened so that we can learn and improve for the next time a similar situation occurs. In the tech world, postmortems are the ultimate tool to learn from server outages and other failures.
Being a software engineer myself, I had to smile when reading about “premortems” for the first time. At that moment, however, I also realized that this seemingly odd term resembles what I do for a living like few other words – but read on.
According to Holiday, this practice – this form of negative visualization – can be attributed to ancient Stoic philosophers:
A writer like Seneca would begin by reviewing or rehearsing his plans, say, to take a trip. And then he would go over, in his head (or in writing), the things that could go wrong or prevent it from happening: a storm could arise, the captain could fall ill, the ship could be attacked by pirates.
Always prepared for disruption, always working that disruption into our plans. Fitted, as they say, for defeat or victory. And let’s be honest, a pleasant surprise is a lot better than an unpleasant one.
Imagining what could go wrong.
Preparing for disruption.
Given my background in web operations and my current job, guess what immediately came to my mind? Call me crazy, but it was the idea of breaking things on purpose – imagining and simulating potential errors in advance – in order to build systems that are resilient to failures.
Your world is ruled by external factors. Promises aren’t kept. […] We are dependent on other people. […] The only guarantee, ever, is that things will go wrong. The only thing we can use to mitigate this is anticipation.
All complex systems will fail. There will always be something that can – and will – go wrong. From self-inflicted outages caused by bad configuration or buggy images to events outside our control like denial-of-service attacks or network failures.
Anticipation is key: hoping for the best, preparing for the worst. (I don’t want to sound overly pessimistic, but this maxim must exist for a reason.)
As a result of our anticipation, we understand the range of potential outcomes and know that they are not all good (they rarely are). We can accommodate ourselves to any of them. We understand that it could possibly all go wrong. And now we can get back to the task at hand. […] We are prepared for failure and ready for success.
By being well prepared, we will not be caught by surprise and won’t be disappointed. And even if there’s nothing we can do about an outage or similar events, we could use that as a practice to manage our expectations. Because sometimes the only way out is through.
Turns out the Stoics knew more about web technology than you’d think.