Mathias Lafeldt2024-03-01T09:58:07+01:00https://sharpend.io/Mathias LafeldtGiving Rust Another Shot in 20202020-07-23T00:00:00+02:00https://sharpend.io/giving-rust-another-shot-in-2020<p>These days I turn to <a href="/recreational-programming-with-serverless/">recreational programming</a> more often than usual. Solving fun problems makes me feel less confused and helpless about the pandemic. It’s a coping mechanism and a good one at that.</p>
<p>One thing has been an especially surprising source of joy for me: <em>Rust</em>. Surprising because I’ve been known to have an on-off relationship with the programming language since its 1.0 release. I quit Rust more than once – both professionally and personally – mainly out of frustration with a subpar development experience and general lack of libraries.</p>
<p>Looking back, I’m thrilled that I gave Rust another chance this year.</p>
<h2 id="that-was-then">That was then</h2>
<p>Back in 2017, <a href="/every-day-we-must-sweep/">I wrote</a> that “Rust lets you control almost everything, but as a beginner, it’s also hard to compile <em>anything</em>. As a result, Rust continues to make me look bad.” I was disappointed. Again.</p>
<p>At the time, I was working for a chaos engineering startup that chose Rust for simulating common server problems such as high CPU load, increased network latency, or random application crashes. A reasonable choice if your goal is to inject low-level faults in a predictable way, right? After all, Rust is said to be <a href="https://thenewstack.io/microsoft-rust-is-the-industrys-best-chance-at-safe-systems-programming/">our best chance at safe systems programming</a>. That may very well be true.</p>
<p>However, back then, I didn’t think Rust was the best option for the client side of the product. The main reason was productivity – or rather the lack thereof.</p>
<p>In 2017, my day-to-day Rust experience boiled down to the following:</p>
<ul>
<li>Writing code in Vim with bare-bones syntax highlighting</li>
<li>No code completion and no “go to definition” (the Rust Language Server was still in its infancy)</li>
<li>Effectively no code formatting, since <code class="language-plaintext highlighter-rouge">rustfmt</code> just gave up whenever a line was too long (seriously)</li>
<li>Docs were hard to find or non-existent</li>
<li>The same goes for high-quality code examples</li>
<li>No official Rust Docker images, homegrown CI pipelines only</li>
<li>Compile times were noticeably slow, resulting in long <a href="/fast-feedback-is-everything/">feedback loops</a></li>
<li>Speaking of feedback, Rust’s borrow checker made it difficult to understand why it rejected a piece of code and how to fix it</li>
</ul>
<p>All these drawbacks add up, but what really bothered me was the immature crate ecosystem. To give you one example, when we implemented the ability to inject faults into containers, we couldn’t find any usable Rust client library to talk to the Docker API, let alone Kubernetes. As a result, we were forced to reinvent the wheel several times (and sometimes ended up with a flat tire, in typical startup fashion).</p>
<p>In the face of these obstacles and despite my appreciation for its language design, I often found it challenging to get real work done with Rust.</p>
<p>At least, that was then.</p>
<p><img src="/assets/images/rust-gegenueber.jpg" alt="" /></p>
<h2 id="this-is-now">This is now</h2>
<p>Today, three years later, I’m happy to report that <a href="https://blog.rust-lang.org/2020/05/15/five-years-of-rust.html">Rust has changed a lot</a>, and for the better! That became apparent to me when I recently had some extra time between freelance contracts and decided to <a href="/7-things-i-learned-from-porting-a-c-crypto-library-to-rust/">port a crypto library from C to Rust</a> (because that’s what I do to get out of the cloud consulting bubble once in a while).</p>
<p>I realized that the release of the <a href="https://blog.rust-lang.org/2018/12/06/Rust-1.31-and-rust-2018.html">2018 edition</a> is particularly noteworthy. It made the language much more productive in several ways: simpler syntax, a smarter borrow-checker, an improved module system, enhanced tooling (both <code class="language-plaintext highlighter-rouge">rustfmt</code> and Clippy, Rust’s linter, hitting 1.0), and a host of other features. Add to that IDE support, incremental compilation, excellent documentation, and immensely helpful error messages, and what you now get is nothing short of remarkable.</p>
<p>Thanks to Rust’s soaring popularity – it has been <a href="https://stackoverflow.blog/2020/01/20/what-is-rust-and-why-is-it-so-popular/">Stack Overflow’s most loved language</a> for four years in a row – finding a particular crate or tool has become easier (though still not necessarily easy). There are more and more <a href="https://github.com/rust-unofficial/awesome-rust">open source projects</a> to explore. There’s a growing list of third-party cargo subcommands (<a href="https://github.com/rust-secure-code/cargo-geiger">cargo-geiger</a> and <a href="https://github.com/gnzlbg/cargo-asm">cargo-asm</a> being two of my favorites). There’s even <a href="https://github.com/google/evcxr">a decent REPL</a>. CI/CD <a href="https://actions-rs.github.io/">is a breeze</a>, and so is <a href="https://github.com/rust-embedded/cross">cross compilation</a> to dozens of platforms. In short, there’s a lot to love here.</p>
<p>Given these advances, it’s no wonder that learning Rust is also quite different now. The tonari blog <a href="https://blog.tonari.no/why-we-love-rust">captured it well</a>:</p>
<blockquote>
<p>A year after switching over to Rust, we onboarded our fourth engineer to the team, who didn’t have much prior experience in either Rust or systems engineering. While the learning curve is undeniable (borrow checker, my old friend), we’ve found that Rust is incredibly empowering for those new to lower-level programming.</p>
</blockquote>
<p><em>Empowering</em> is the perfect word to describe Rust in 2020. What used to be a rough adventure with many pitfalls has turned into something beautiful, something that can lift your spirit. At least, that’s what it did for me.</p>
<p>I encourage you to immerse yourself in a little Rust project today. You might be in for a pleasant surprise.</p>
<p>Update: <a href="https://www.reddit.com/r/rust/comments/hwdaxo/giving_rust_another_shot_in_2020/">Reddit thread</a></p>
7 Things I Learned From Porting a C Crypto Library to Rust2020-07-01T00:00:00+02:00https://sharpend.io/7-things-i-learned-from-porting-a-c-crypto-library-to-rust<p>Rust has always been the programming language that reminds me the most of <a href="/what-i-learned-from-hacking-video-games/">my game hacking days</a>, and for good reasons. Rust is a natural fit for embedded systems like video game consoles – or rather <a href="https://gitlab.com/flio/rustation-ng">emulators</a> thereof. The compiler supports <a href="https://forge.rust-lang.org/release/platform-support.html">a high number of platforms</a> and lets you drop down to C or assembly if necessary. Plus, the language lends itself well to implementing (reverse-engineered) cryptographic algorithms and other low-level shenanigans.</p>
<p>Given these benefits, it’s no surprise that I keep coming back to Rust. This time, I decided to revisit <a href="https://github.com/mlafeldt/cb2util/pull/13">a pull request</a> from five years ago, which has been lingering in my mind ever since. The ambiguous goal of the pull request is to port cb2util, one of my old crypto tools for PlayStation 2, from C to pure Rust.</p>
<p>Among other things, cb2util allows you to decrypt all cheat codes for the notorious <a href="https://en.wikipedia.org/wiki/Code_Breaker">CodeBreaker</a> cheat device, making it possible to scrutinize them in their raw form and use them with other devices.</p>
<p>To give you an idea, the following code might make your character invincible by constantly writing the byte value 99 (0x63) to memory address 0x0096F5B8:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Infinite health
0096F5B8 00000063
</code></pre></div></div>
<p>The very same code but encrypted:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Infinite health
81E1C95B 9764DA87
</code></pre></div></div>
<p>Back in 2006, it took me weeks to reverse-engineer the proprietary encryption scheme from MIPS 5900 assembly and <a href="https://github.com/mlafeldt/cb2util/blob/v1.9/cb2_crypto.c">convert everything to C</a> in a piecemeal fashion. Naturally, I learned a ton in the process. Trying to port those same crypto routines from C to Rust now, 14 years later, sounded like another promising learning opportunity.</p>
<p>Luckily, I wasn’t disappointed. In fact, I went so far as to extract and publish everything that I did as a Rust crate. It’s called, well, <a href="https://github.com/mlafeldt/codebreaker-rs">codebreaker</a>.</p>
<p>Here are seven things I picked up while working on this little side project – some of them specific to cryptography, others more general in nature.</p>
<h2 id="soft-migration-via-ffi">Soft migration via FFI</h2>
<p>Rust provides a <a href="https://blog.rust-lang.org/2015/04/24/Rust-Once-Run-Everywhere.html">foreign function interface</a> (FFI) to talk to other programming languages with ease. This interface allowed me to <a href="https://github.com/mlafeldt/cb2util/pull/13/commits/1618f7e6a120c1fb8d49f1dd5d0ebc39861655ff">gradually port</a> one C function at the time until there was no foreign function call left. Since the crypto code involves a state machine, I also had to <a href="https://github.com/mlafeldt/cb2util/pull/13/commits/03f63639f793d800720bc69fd43ea3a182b9ffc8">temporarily export</a> a few global C variables in order to access them from Rust. A large set of existing <a href="https://github.com/mlafeldt/cb2util/tree/v1.9/test">integration tests</a> ensured that I didn’t break anything along the way, which was very helpful (thanks, past me!).</p>
<h2 id="wrapping-operations">Wrapping operations</h2>
<p>In C, unsigned integer arithmetic is defined to be modulus a power of two, which is almost always what you want when it comes to cryptography. To achieve the same in Rust without running into any overflow errors, you have to use dedicated wrapping operations for modular addition (<code class="language-plaintext highlighter-rouge">wrapping_add</code>), subtraction (<code class="language-plaintext highlighter-rouge">wrapping_sub</code>), multiplication (<code class="language-plaintext highlighter-rouge">wrapping_mul</code>), etc.</p>
<p>Here’s an example for <a href="https://doc.rust-lang.org/std/primitive.u32.html">32-bit unsigned integers</a>:</p>
<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nd">assert_eq!</span><span class="p">(</span><span class="mi">200u32</span><span class="nf">.wrapping_add</span><span class="p">(</span><span class="mi">55</span><span class="p">),</span> <span class="mi">255</span><span class="p">);</span>
<span class="nd">assert_eq!</span><span class="p">(</span><span class="mi">200u32</span><span class="nf">.wrapping_add</span><span class="p">(</span><span class="nn">u32</span><span class="p">::</span><span class="n">MAX</span><span class="p">),</span> <span class="mi">199</span><span class="p">);</span>
</code></pre></div></div>
<h2 id="transmuting-types">Transmuting types</h2>
<p>Due to the nature of the encryption scheme, it’s sometimes necessary to read or write the same data in two different ways. For example, I defined an <a href="https://github.com/mlafeldt/codebreaker-rs/blob/0.2.1/src/rc4.rs">RC4</a> key as a fixed-size array <code class="language-plaintext highlighter-rouge">[u32; 5]</code>, which is convenient most of the time. That is, until the key itself needs to be encrypted as a slice of type <code class="language-plaintext highlighter-rouge">&[u8]</code>.</p>
<p>Doing that in C via pointer casting is easy enough, but it took me a while to figure out how to implement it in Rust. I found one (albeit unsafe) solution in the <a href="https://github.com/BurntSushi/byteorder/blob/c0143a7a07a900d186dde205e08134fedfa5a9e3/src/io.rs#L1589-L1599">byteorder</a> crate for transmuting types – reinterpreting the bits of a value of one type as another type:</p>
<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">unsafe</span> <span class="k">fn</span> <span class="n">slice_to_u8_mut</span><span class="o"><</span><span class="n">T</span><span class="p">:</span> <span class="nb">Copy</span><span class="o">></span><span class="p">(</span><span class="n">slice</span><span class="p">:</span> <span class="o">&</span><span class="k">mut</span> <span class="p">[</span><span class="n">T</span><span class="p">])</span> <span class="k">-></span> <span class="o">&</span><span class="k">mut</span> <span class="p">[</span><span class="nb">u8</span><span class="p">]</span> <span class="p">{</span>
<span class="k">use</span> <span class="nn">std</span><span class="p">::</span><span class="nn">mem</span><span class="p">::</span><span class="n">size_of</span><span class="p">;</span>
<span class="k">let</span> <span class="n">len</span> <span class="o">=</span> <span class="nn">size_of</span><span class="p">::</span><span class="o"><</span><span class="n">T</span><span class="o">></span><span class="p">()</span> <span class="o">*</span> <span class="n">slice</span><span class="nf">.len</span><span class="p">();</span>
<span class="nn">slice</span><span class="p">::</span><span class="nf">from_raw_parts_mut</span><span class="p">(</span><span class="n">slice</span><span class="nf">.as_mut_ptr</span><span class="p">()</span> <span class="k">as</span> <span class="o">*</span><span class="k">mut</span> <span class="nb">u8</span><span class="p">,</span> <span class="n">len</span><span class="p">)</span>
<span class="p">}</span>
</code></pre></div></div>
<p>Update: Thanks to Reddit user DannoHung, I switched to using <a href="https://crates.io/crates/bytemuck">bytemuck</a> for transmuting. See <a href="https://github.com/mlafeldt/codebreaker-rs/pull/2">this pull request</a> for details.</p>
<h2 id="rsa-with-num-bigint">RSA with num-bigint</h2>
<p>In addition to RC4, I needed a simple RSA implementation for small 64-bit keys (the size of one cheat code). I used to rely on <a href="https://github.com/mlafeldt/libbig_int">libbig_int</a>, a portable C library to calculate integers of arbitrary length. Now I needed something similar for Rust. That something turned out to be the superb <a href="https://crates.io/crates/num-bigint">num-bigint</a> crate.</p>
<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">fn</span> <span class="nf">rsa_crypt</span><span class="p">(</span><span class="n">addr</span><span class="p">:</span> <span class="o">&</span><span class="k">mut</span> <span class="nb">u32</span><span class="p">,</span> <span class="n">val</span><span class="p">:</span> <span class="o">&</span><span class="k">mut</span> <span class="nb">u32</span><span class="p">,</span> <span class="n">rsakey</span><span class="p">:</span> <span class="nb">u64</span><span class="p">,</span> <span class="n">modulus</span><span class="p">:</span> <span class="nb">u64</span><span class="p">)</span> <span class="p">{</span>
<span class="k">let</span> <span class="n">code</span> <span class="o">=</span> <span class="nn">BigUint</span><span class="p">::</span><span class="nf">from_slice</span><span class="p">(</span><span class="o">&</span><span class="p">[</span><span class="o">*</span><span class="n">val</span><span class="p">,</span> <span class="o">*</span><span class="n">addr</span><span class="p">]);</span>
<span class="k">let</span> <span class="n">m</span> <span class="o">=</span> <span class="nn">BigUint</span><span class="p">::</span><span class="nf">from</span><span class="p">(</span><span class="n">modulus</span><span class="p">);</span>
<span class="k">if</span> <span class="n">code</span> <span class="o"><</span> <span class="n">m</span> <span class="p">{</span>
<span class="k">let</span> <span class="n">digits</span> <span class="o">=</span> <span class="n">code</span><span class="nf">.modpow</span><span class="p">(</span><span class="o">&</span><span class="nn">BigUint</span><span class="p">::</span><span class="nf">from</span><span class="p">(</span><span class="n">rsakey</span><span class="p">),</span> <span class="o">&</span><span class="n">m</span><span class="p">)</span><span class="nf">.to_u32_digits</span><span class="p">();</span>
<span class="o">*</span><span class="n">addr</span> <span class="o">=</span> <span class="n">digits</span><span class="p">[</span><span class="mi">1</span><span class="p">];</span>
<span class="o">*</span><span class="n">val</span> <span class="o">=</span> <span class="n">digits</span><span class="p">[</span><span class="mi">0</span><span class="p">];</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre></div></div>
<h2 id="no-stdlib">No stdlib</h2>
<p>Speaking of num-bigint, I studied its sources intensively and <a href="https://github.com/mlafeldt/codebreaker-rs/pull/1">learned how to write code</a> that can also be used for embedded environments such as game consoles (another CodeBreaker clone, anyone?). Said code must <em>not</em> depend on Rust’s standard library. This exercise showed me how straightforward conditional compilation and optional dependencies are in Rust.</p>
<h2 id="mod_inverse-using-newtons-method">mod_inverse using Newton’s method</h2>
<p>As the saying goes, “make it work, make it right, make it fast”. After porting all crypto routines to Rust, I did some local optimizations. One of them involved replacing a messy reverse-engineered function with something much more mathematically sound. As luck would have it, I stumbled upon <a href="https://lemire.me/blog/2017/09/18/computing-the-inverse-of-odd-integers/">a blog post by Daniel Lemire</a> that describes an elegant method for computing the multiplicative inverse of odd integers. I proudly present <a href="https://github.com/mlafeldt/codebreaker-rs/commit/6431adcb051861336d8a49c421d74b56583c3729">the diff</a>:</p>
<p><img src="/assets/images/rust-mod-inverse.png" alt="" /></p>
<h2 id="automated-testing-with-actions-rs">Automated testing with actions-rs</h2>
<p>Being the diligent engineer that I am, I devoted some time to unit tests and doctests (executable examples in the documentation). I also became friends with <a href="https://github.com/rust-lang/rust-clippy">Clippy</a>, Rust’s amazing linter, which helped me write code that’s more correct, more readable, and more idiomatic. Finally, I brought everything together in a handy <a href="https://github.com/mlafeldt/codebreaker-rs/blob/0.2.1/.github/workflows/rust.yaml">CI pipeline</a> using <a href="https://actions-rs.github.io/">actions-rs</a>, the GitHub Actions toolkit for Rust.</p>
<hr />
<p>All in all, I can confidently say that the Rust port is better than the original in many ways. <a href="https://github.com/mlafeldt/codebreaker-rs">But see for yourself</a>.</p>
<p>If you have any additions or questions, feel free to ping me on Twitter (<a href="https://twitter.com/mlafeldt">@mlafeldt</a>). I’m always eager to learn something new! ✌️</p>
<p>Update: <a href="https://www.reddit.com/r/rust/comments/hj9pvu/7_things_i_learned_from_porting_a_c_crypto/">Really good Reddit thread</a></p>
Recreational Programming with Serverless2020-06-08T00:00:00+02:00https://sharpend.io/recreational-programming-with-serverless<p>This article is a slightly edited version of the presentation I gave at the <a href="https://www.aws-community-day.de/">AWS Community Day 2019</a> in Hamburg, Germany. You can find the original slides <a href="https://speakerdeck.com/mlafeldt/recreational-programming-with-serverless">here</a>. There’s no recording available, so you have to take my word that the following actually happened.</p>
<p>I know it’s been nine months since the event took place, but I still find myself thinking about the importance of recreational programming, especially in difficult times like this. That’s why I decided to share the full content here and now. Besides, following the “show, don’t tell” rule, I’ve recently <a href="https://github.com/mlafeldt/dilbert-feed">open sourced the serverless project</a> that is at the heart of the talk.</p>
<p>One more note before we get started: If I appear overly critical of Kubernetes at times, it’s because I tell the story from the perspective of someone who’s <em>built and operated</em> container systems in different cloud environments since 2015. Mere users of Kubernetes will undoubtedly appreciate it more than I do. <a href="https://ferd.ca/complexity-has-to-live-somewhere.html">Complexity has to live somewhere</a>, after all.</p>
<p>In any case, enjoy the presentation.</p>
<hr />
<p><img src="/assets/images/recreational-programming/01.png" alt="" /></p>
<p>Hello, and welcome to my lightning talk on recreational programming with serverless!</p>
<p><img src="/assets/images/recreational-programming/02.png" alt="" /></p>
<p>I want to start with a little introduction. My name is Mathias Lafeldt. I’m from a small town north of Hamburg, so I’m a local if you will.</p>
<p>I’ve been using AWS since 2013. These days, I help companies embrace the cloud as a freelancer – <a href="/freelancer-by-choice/">a very happy freelancer</a>, to be honest.</p>
<p><img src="/assets/images/recreational-programming/03.png" alt="" /></p>
<p>Most of my current consulting work focuses on Kubernetes and AWS. I help teams set up AWS accounts in a sustainable way, do network planning, pick the right cluster setup for them, create CI/CD pipelines, implement monitoring, etc. – basically, all those things that are required to make any use of Kubernetes on AWS.</p>
<p>As a freelancer, I see a high demand for migrating software to the cloud. A lot of organizations choose Kubernetes for modernizing their legacy applications. I understand the reasoning behind this decision:</p>
<ul>
<li>It’s relatively easy to package your code and move it (almost as-is) to the cloud in the form of containers.</li>
<li>Kubernetes streamlines the development process by automating the deployment and scaling of containerized applications.</li>
</ul>
<p>I think it’s true that Kubernetes <em>can</em> provide value. However…</p>
<p><img src="/assets/images/recreational-programming/04.png" alt="" /></p>
<p>It’s also true that Kubernetes is a complex beast with many moving parts, and therefore many different failure modes. To make matters worse, it’s growing by the minute as its ecosystem is constantly evolving.</p>
<p>You may know <a href="https://k8s.af/">this website</a>. It was created by Henning Jacobs from Zalando. He built this collection of Kubernetes failure stories for people to learn from each other and not repeat the same mistakes again and again.</p>
<p>For me, this site’s sheer existence is a testament to the fact that there’s <strong>a lot</strong> to deal with when you want to be successful with Kubernetes in production.</p>
<p><img src="/assets/images/recreational-programming/05.png" alt="" /></p>
<p>I like to joke that (self-hosted) Kubernetes clusters are the perfect job creation measure. They’re also well suited if you want to get good at <a href="/writing-your-first-postmortem/">writing postmortems</a>. I know it first-hand, as I had to write a couple of those myself…</p>
<p>Now you may ask, what about EKS, Amazon’s managed Kubernetes service? [The talk right before mine was about EKS.]</p>
<p>From my experience, EKS offers significant advantages over tools like <code class="language-plaintext highlighter-rouge">kops</code>. For one, it will give you a managed control plane that takes care of running the API servers and <code class="language-plaintext highlighter-rouge">etcd</code> for you. However, you still need to bring your own EC2 worker nodes, which means you’re still responsible – at least to a degree – for high availability, AMI updates, EBS volumes, backups, VPC design, load balancers, and much more.</p>
<p>[Note: AWS has released <a href="https://aws.amazon.com/blogs/containers/eks-managed-node-groups/">EKS managed node groups</a> in the meantime, which certainly reduces the total cost of ownership (TCO) of Kubernetes clusters, especially when it comes to updating worker nodes. The <em>overall TCO</em> nevertheless continues to represent a significant administrative burden on the shoulders of operators. <a href="https://github.com/kubernetes/community/blob/ddc11a75837b29498f120529ca57145f4a242cbc/wg-security-audit/findings/Kubernetes%20Final%20Report.pdf">Configuration of Kubernetes remains a non-trivial task</a>.]</p>
<p><a href="https://unsplash.com/photos/ZMcLVBi9xx4"><img src="/assets/images/recreational-programming/06.png" alt="" /></a></p>
<p>While Kubernetes might be perfect for you or your company (understandably so given its advantages), I personally find working with it for extended periods exhausting. As a cluster operator, it’s stressful to be on the hook for so many things that can go wrong – and will go wrong, as demonstrated by the failure stories website mentioned earlier.</p>
<p>Ultimately, it should be about the <em>applications</em>, right? We don’t have a Kubernetes cluster just for the sake of having a Kubernetes cluster. Its whole purpose is to make it easy for us to deploy, scale, and manage our containerized applications.</p>
<p>Sadly, despite considerable efforts, we still get sucked into a never-ending cycle of operational tasks, tasks we never signed up for. The result for me has been mental fatigue and, at times, lack of motivation to deal with such systems.</p>
<p><img src="/assets/images/recreational-programming/07.png" alt="" /></p>
<p>At some point, I discovered a <a href="https://corecursive.com/025-burn-out-and-recreational-programming/">podcast episode</a> with Jamis Buck. Jamis is a famous programmer. He created Capistrano and plenty more open source projects. On the show, he talks about how he used to be on the top of the world about ten years ago. He worked for 37signals (now called Basecamp), earned a big paycheck, everything seemed perfect. There was only one problem: he was burnt out.</p>
<p>As a consequence, he had to let go of most of his side projects. Unfortunately, that didn’t help much. To overcome burnout, he left his fantastic job and decided to write a book on one of his passions: generating mazes. Mazes helped Jamis remember what got him excited about programming initially. Mazes are his form of <strong>recreational programming</strong>.</p>
<p>I was wondering what my kind of recreational programming would be? Maybe I already found it but didn’t know it yet.</p>
<p><img src="/assets/images/recreational-programming/08.png" alt="" /></p>
<p>Werner Vogels recently published a blog post titled <a href="https://www.allthingsdistributed.com/2019/08/modern-applications-at-aws.html">“Modern applications at AWS”</a>. In it, he describes how Amazon has been successful for 20 years by going through a series of radical transformations, always questioning how they build applications and how they organize the company.</p>
<p><img src="/assets/images/recreational-programming/09.png" alt="" /></p>
<p>According to Werner, organizations must adopt five elements to increase agility and innovation speed:</p>
<ol>
<li>Embrace microservices to decouple systems and enable autonomous, cross-functional teams.</li>
<li>Use purpose-built databases for each microservice rather than a single database for all microservices, which can’t meet specific needs and is a single point of failure.</li>
<li>Enable teams to release changes independently, for example, by providing best-practice infrastructure-as-code templates.</li>
<li>The same goes for security as “in modern applications, security features are built into every component of the application and automatically tested and deployed with each release.”</li>
<li><strong>Be as serverless as possible and offload undifferentiated tasks to AWS services such as Lambda.</strong></li>
</ol>
<p><img src="/assets/images/recreational-programming/10.png" alt="" /></p>
<p>Even Amazon is not completely serverless yet, but they’re getting there. Werner believes that thanks to serverless, there will soon be a whole generation of developers who have never touched a server and only write business logic.</p>
<p>That sounds like a bright future to me. ☀️</p>
<p><img src="/assets/images/recreational-programming/11.png" alt="" /></p>
<p>Of course, there’s more to serverless than programming with Lambda. For starters, it’s not just Lambda but the entire application stack, including services like DynamoDB, S3, SNS, API Gateway, etc. More generally speaking:</p>
<ul>
<li>Serverless services are managed services that run without the need for infrastructure provisioning and scaling.</li>
<li>They provide built-in availability and security. No need to care about availability zones or kernel patches.</li>
<li>You only pay for what you use. You don’t pay for idle resources.</li>
<li>Serverless allows you to focus on business logic – your “secret sauce”, the things that set you apart from your competition.</li>
<li>With serverless, you can create value for customers faster.</li>
</ul>
<p>Looking at this list made me pause for a second. Those are indeed <em>excellent reasons</em> for startups and enterprises to go serverless.</p>
<p><img src="/assets/images/recreational-programming/12.png" alt="" /></p>
<p>More to the point, I realized that a lot of properties that make serverless great for businesses – no servers to manage, easy deployments, pay-as-you-go with <a href="https://aws.amazon.com/free/">a generous free tier</a> – also make it a great fit for recreational programming!</p>
<p>One advantage stands out in particular to me: the ability to concentrate on my applications – the serious ones and the not-so-serious ones.</p>
<p><img src="/assets/images/recreational-programming/13.png" alt="" /></p>
<p>Next, I want to show you a few serverless projects I created for fun, projects I consider recreational in some sense.</p>
<p>I’m a huge fan of <a href="https://dilbert.com/">Dilbert</a> and a couple of other comic strips. I also love reading articles and comics using my favorite RSS feed reader: Feedly on iOS.</p>
<p><img src="/assets/images/recreational-programming/14.png" alt="" /></p>
<p>The problem with Dilbert is that although there’s an official RSS feed, it no longer includes the comic strips themselves but only links to dilbert.com. How mean and inconvenient!</p>
<p>In 2017, I started looking at this obstacle as an opportunity for an interesting serverless application – bluntly named <a href="https://github.com/mlafeldt/dilbert-feed">dilbert-feed</a> – to create my own feed that I can enjoy in Feedly again.</p>
<p>Here’s how I went about it.</p>
<p><img src="/assets/images/recreational-programming/15.png" alt="" /></p>
<p>First of all, I had to figure out how to get the images. Fortunately, Dilbert is a daily strip and the URLs are predictable, e.g., <code class="language-plaintext highlighter-rouge">dilbert.com/strip/2019-08-30</code>. That means I only had to download the web page for a given day and parse the HTML to get a link to the image.</p>
<p>I decided to use Go – my favorite programming language – for the job. I was lucky to find the superb <a href="https://github.com/PuerkitoBio/goquery">goquery</a> package, which does most of the work for me, as you can see on the slide above.</p>
<p><img src="/assets/images/recreational-programming/16.png" alt="" /></p>
<p>Now that I knew how to get the images, I used the <a href="https://www.serverless.com/">Serverless Framework</a> to turn that code, which has to be deployed somewhere where it can be invoked once a day, into a Lambda function called <code class="language-plaintext highlighter-rouge">get-strip</code>. After determining the image URL, the function will copy the found image to an S3 bucket via the always handy <a href="https://github.com/aws/aws-sdk-go">AWS SDK for Go</a>.</p>
<p>To separate concerns, I wrote another Lambda named <code class="language-plaintext highlighter-rouge">gen-feed</code> that generates the RSS feed for the last 30 days (for this, all it needs to know is the location of the images uploaded by <code class="language-plaintext highlighter-rouge">get-strip</code>).</p>
<p><img src="/assets/images/recreational-programming/17.png" alt="" /></p>
<p>For the feed to be consistent (and to make the task a bit more challenging), the two functions should run in sequence. But instead of staggering the Lambdas via two cron jobs, I chose to give <a href="https://aws.amazon.com/step-functions/">AWS Step Functions</a> a spin.</p>
<p>What you see on this slide must be one of the simplest state machines imaginable. While Step Functions is much more powerful, bare-bones orchestration was all I needed to get started.</p>
<p><img src="/assets/images/recreational-programming/18.png" alt="" /></p>
<p>To complete the picture, I created a CloudWatch Events rule for triggering the state machine to update the feed with the latest Dilbert strip every morning. The architecture diagram shows all involved components running inside my personal AWS account (for which I haven’t had to pay a cent so far, by the way).</p>
<p>You’re probably right to assume that dilbert-feed is over-engineered to some degree, and deliberately so. Remember that it’s just a fun side project, a playground where I can do whatever I want and try out new tools and practices whenever I feel like it.</p>
<p>Among other things, I used the project to explore different monitoring/observability solutions for serverless. In the end, I settled for a simple <code class="language-plaintext highlighter-rouge">heartbeat</code> Lambda that pings <a href="https://healthchecks.io/">Healthchecks.io</a> over HTTP as the last state machine step.</p>
<p><img src="/assets/images/recreational-programming/19.png" alt="" /></p>
<p>On Healthschecks.io, I’ve configured a check that sends me an email notification as soon as a ping doesn’t arrive on time. It can’t get much easier than that.</p>
<p>In true <em>deploy-once-and-leave-the-rest-to-serverless</em> manner, the setup has been humming along nicely for over two years without significant problems other than dilbert.com being down for maintenance. 💪</p>
<p><img src="/assets/images/recreational-programming/20.png" alt="" /></p>
<p>As you can see from this list, I learned a great deal from hacking on dilbert-feed on nights and weekends. In fact, it continues to benefit me as a playground and template for other side and freelance projects to this day.</p>
<p>[Shortly after giving the presentation in 2019, I started to embrace the wonderful <a href="https://aws.amazon.com/cdk/">AWS CDK</a>. Again, all these experiments <a href="https://github.com/mlafeldt/dilbert-feed">are open source</a>.]</p>
<p><img src="/assets/images/recreational-programming/21.png" alt="" /></p>
<p>Before we wrap up, I want to briefly mention two more serverless projects I made for fun.</p>
<p>The first is a <a href="https://github.com/mlafeldt/launchdarkly-dynamo-store">DynamoDB Store for LaunchDarkly</a>, which provides the building blocks that, taken together, allow you to create a serverless flag storage pipeline. For more information, check out my presentation on <a href="https://speakerdeck.com/mlafeldt/implementing-feature-flags-in-serverless-environments">Implementing Feature Flags in Serverless Environments</a>.</p>
<p>Needless to say, this was a good opportunity for diving into DynamoDB.</p>
<p><img src="/assets/images/recreational-programming/22.png" alt="" /></p>
<p>Last but not least, I’ve been tinkering with a serverless version of Chaos Monkey. It’s still work-in-progress, but I hope to share more about it in the future. Suffice to say, <a href="/chaos-engineering-101/">Chaos Engineering</a> is near and dear to my heart.</p>
<p><img src="/assets/images/recreational-programming/loopy.gif" alt="" class="align-center" /></p>
<p>What do all these ventures have in common? They’re part of a <em>positive feedback</em> loop.</p>
<p>Investing time in side projects – or recreational programming in general, the “project” bit is optional – will improve my freelance work, either directly (craftsmanship) or indirectly (<a href="/blog/what-motivates-me/">motivation</a>). Conversely, the things I learn from consulting can have a regenerative effect on my side projects. It’s a virtuous cycle.</p>
<p>(That said, it’s totally okay and often advantageous if your hobbies have nothing to do with your work. I can only speak for myself.)</p>
<p><img src="/assets/images/recreational-programming/24.png" alt="" /></p>
<p>Let’s wrap things up. Here are the key takeaways for you:</p>
<p>Cloud projects don’t have to be big or great. Sometimes, it’s enough for them to be fun. For me, serverless is the very definition of fun.</p>
<p>Building things is fulfilling; servers are a distraction from what really matters: our beloved applications.</p>
<p>Serverless has helped me, a consultant who wrestles with Kubernetes by day, rediscover the joy of programming by night.</p>
<p><strong>Serverless is an excellent choice for many endeavors, one of them being recreational programming.</strong></p>
<p><img src="/assets/images/recreational-programming/25.png" alt="" /></p>
<p>Thank you. 🙏</p>
Freelancer by Choice2020-02-24T00:00:00+01:00https://sharpend.io/freelancer-by-choice<p><img src="/assets/images/choice.jpg" alt="" /></p>
<p>I’m a <em>freelancer</em>. To be more precise, I’ve been a freelancer for three years now.</p>
<p>I had never planned to escape the traditional 9-to-5 office job to work for myself, though. For ten years, I genuinely enjoyed being a full-time employee at a handful of different tech companies. I took on various roles from software developer to systems engineer to site reliability engineer. Life was good, <a href="/blog/what-motivates-me/">sometimes great</a>, following an ever-increasing up-and-to-the-right trajectory – or at least it felt that way.</p>
<p>The truth is, becoming a freelancer was initially just a side effect. You might go so far as to call it an accident. Both are true.</p>
<p>In March 2017, after two months of back and forth (I loved the job I had at the time), I decided to jump in the deep end and join a Silicon Valley startup as employee #2. Well, technically, I was just a remote contractor based in Germany. Regular employment was not an option as the company didn’t have an office in the EU. So I ended up signing a consulting agreement, submitting monthly invoices for my work – mostly Rust & Java programming, AWS infrastructure automation, and some technical writing. For this whole arrangement to work with reasonable effort, I had to go freelance.</p>
<p>Since becoming and being a freelancer in Germany involves a fair amount of <a href="https://www.settle-in-berlin.com/how-to-become-a-freelancer-in-germany-self-employed/">bureaucracy</a>, I decided early on that getting a good accountant would be a make-or-break deal for me. While I have a basic understanding of taxes, I couldn’t care less about the intricacies of accounting. I’d rather spend my time doing things I enjoy. Luckily, a freelance friend of mine recommended a competent firm, which has taken good care of all the paperwork so far.</p>
<p>The startup gig ultimately turned out to be a rather forgettable experience. I quit after only eight months. But even though it didn’t work out as planned, I don’t have any regrets – at least not today, with the benefit of hindsight. On the one hand, I learned a lot about startup culture, remote work, <a href="/the-limitations-of-chaos-engineering/">Chaos Engineering</a>, and the business of SaaS. On the other hand, I also learned a lot – if not more – about <em>myself</em>:</p>
<ul>
<li>I can work for myself. I’m capable of accomplishing things on my own. I somehow knew this before the startup job, but managing myself with the rest of the team nine time zones away was quite a different kind of challenge.</li>
<li>I’m a self-driven person, always eager to learn more and comfortable wearing multiple hats. And yet I also know that <a href="/every-day-we-must-sweep/">my ego can get in the way</a> at times.</li>
<li>I’ve become an <a href="https://twitter.com/mlafeldt/status/1202048151802089472">AWS expert</a> who can program and write. That’s a precious combination of skills. As a result, I can charge top dollar for my hard-earned knowledge.</li>
<li>During my career, I’ve built a network of awesome people I can tap into for help on all sorts of issues ranging from technical questions to legal and business affairs.</li>
<li>I used to be too risk-averse. I blame it in part on growing up in former East Germany and not having entrepreneurial DNA in my family (it’s more complicated than that, of course). Fortunately, I’ve learned to be braver over time. Contracting for a startup was a bold move for me – and so was my early resignation.</li>
</ul>
<p>So what did I do with this new-found awareness? I’d be lying if I said I didn’t go looking for another <em>permanent</em> job at first. I did – likely out of habit and laziness. However, nothing came out of it except for a dozen unexceptional interview experiences (yes, <a href="https://www.jarednelsen.dev/posts/The-horrifically-dystopian-world-of-software-engineering-interviews">hiring is broken</a>, but that’s not the point). It was frustrating and confusing, and it somehow didn’t feel right. I found myself struggling with my professional identity, facing questions I couldn’t answer easily: Who am I? How do I want to work? Employee or freelancer? And what about doing my own thing and becoming a co-founder, which was also an option at the time?</p>
<p>At some point, after weeks of contemplating, it struck me that I was actually given a rare opportunity. I realized that the startup gig was not, in fact, a failure, but rather the gentlest introduction to freelancing I could’ve asked for. Come to think of it, I learned most things a “normal” freelancer should know – without the unpleasant task of finding new clients (which, admittedly, is a critical skill to pick up sooner rather than later). I began to see the situation in a new light – as a stepping stone between full-time employment and being a freelancer, this time for real.</p>
<p>My first client came through a former coworker. So did the second and the third. To be honest, nearly all client relationships have started this way. I know someone (who knows someone) who trusts me to do great work. That’s where the network of wonderful people I mentioned earlier comes in.</p>
<p>What exactly do I do now? As a freelance cloud consultant, I specialize in designing and building robust serverless and container-based solutions for clients who want to take their businesses to the cloud. (<a href="/about/">Read more about my work here</a>.)</p>
<p>The one thing I like the most about freelancing is, unsurprisingly, the incredible amount of freedom it offers. Scott Berkun perhaps <a href="https://scottberkun.com/2015/my-creative-burnout/">put it best</a>: “My chosen career requires some significant sacrifices, but a major benefit is on most days I answer to no one. When I need time for myself it’s there for me to take it.” Many employees would be surprised just how much freedom and flexibility this way of working can provide. In fact, it still astonishes <em>me</em>.</p>
<p>All things considered, I’m a happy consultant, grateful for the opportunities that have come my way so far – including the bad ones as they make the good ones possible. Granted, I didn’t have to survive lots of ups, downs, and evolutions of my business. I know I’m still winging it in some areas, but that’s okay. Freelancing can be a stable, long-term career. To quote Paul Jarvis, “As long as you’re doing great work that’s in demand, working for yourself has <em>no limits</em>.”</p>
<p>The new normal is that I work for myself. I’m proud to be a freelancer by choice.</p>
The Limitations of Chaos Engineering2018-01-03T00:00:00+01:00https://sharpend.io/the-limitations-of-chaos-engineering<p><img src="/assets/images/fireman.jpg" alt="" /></p>
<p><a href="https://principlesofchaos.org/">Chaos Engineering</a>, and fault injection in particular, is all the rage. Breaking things on purpose, rather than “by accident”, is what the cool kids do these days. It’s what they like to speak about at meetups and conferences or proudly promote on Twitter and their blogs. We’re starting to see job titles with the word “Chaos” in them, not unlike “DevOps” a couple of years ago (both meaningless, of course). Sarcasm aside, it’s evident that Chaos Engineering has become a <a href="https://www.thoughtworks.com/radar/techniques/chaos-engineering">technology trend</a>, with more and more companies adopting it. While it might not have gone mainstream yet, we’re getting there for sure!</p>
<p>I’ve certainly contributed to the hype myself by <a href="/articles">publishing articles</a>, <a href="/talks">giving presentations</a>, and <a href="https://github.com/mlafeldt">developing tools</a> in the Chaos Engineering space. I even had a brief stint at a startup trying to sell “Failure as a service” – or “Resilience as a service”, one of the two – to enterprises. All in all, I think it’s fair to say that I’m an avid proponent of the practice. I believe that <em>every</em> Site Reliability Engineer should, at the very least, be familiar with the basics of proactive failure testing as a means to create better systems. (There’s an “R” in SRE for a reason.)</p>
<p>But here’s the rub: Being an advocate of something doesn’t mean you should close your eyes to its downsides and limitations. In fact, the most skilled engineers are well aware of the pros <em>and</em> cons of their favorite tool or method, and consider them carefully. I have the impression, however, that most discussions so far have focused almost exclusively on the advantages of Chaos Engineering – occasionally in a hyperbolic manner – without asking a lot of hard questions.</p>
<p>I, too, am guilty of this, as are many other Practitioners of Chaos (we definitely have the coolest names). This article is my attempt to take off those rose-colored glasses, at least for a moment, and put things into perspective. Believe it or not, Chaos Engineering does have its gotchas and limitations.</p>
<h2 id="a-means-to-an-end">A Means to an End</h2>
<p>First of all, it should be clear that Chaos Engineering is a means to an end, not an end in itself. Experimenting on a distributed system is of great worth, but what matters, in the end, is the <em>production service</em> you aim to improve in the first place. Breaking things is a ton of fun, I can attest to that, but as long as you don’t feed results back – by fixing flaws, tweaking runbooks, training people – your chaos experiments are rarely more than a time killer. And no, updating your mental models isn’t enough (that is unless you never forget anything and everyone has access to your brain). At the very least, write down any observations you make and follow up soon, if required.</p>
<p><strong>Fault injection on its own won’t make your infrastructure more robust; people will.</strong> It should be obvious, but it’s not. Last year I reviewed an early draft of a book on Chaos Engineering. I was surprised to learn that there was no mention of any steps beyond the experimentation phase, as if that was the ultimate goal.</p>
<p>(A quick note: I’m well aware that fault injection and Chaos Engineering are <em>not</em> the same. When the latter comes up in practice, however, people almost always talk about inducing faults into distributed systems as an opportunity to learn, so cut me some slack here.)</p>
<h2 id="one-step-forward-two-steps-back">One Step Forward, Two Steps Back</h2>
<p>Even if you do the work, who’s to say that being able to uncover weaknesses will automatically lead to positive outcomes, like improved customer experience? As software developers know, identifying a bug and fixing it are two different challenges. Indeed, your optimization efforts in one area might increase brittleness in other areas, as <a href="https://www.researchgate.net/publication/276139783">David Woods</a> points out:</p>
<blockquote>
<p>[Expanding] a system’s ability to handle some additional perturbations, increases the systems vulnerability in other ways to other kinds of events. This is a fundamental trade-off for complex adaptive systems where becoming more optimal with respect to some variations, constraints, and disturbances increases brittleness in the face of variations, constraints, and disturbances that fall outside this set.</p>
</blockquote>
<h2 id="one-among-many">One Among Many</h2>
<p>Chaos Engineering is not a remedy for all of your reliability concerns, and it never will be. It’s merely <a href="https://queue.acm.org/detail.cfm?id=2889274">one of many approaches</a> used to gain confidence in system correctness (typically in the face of perturbation). Consider it required but not sufficient. And by no means is it – or should it be – the only way to learn from failure. As John Allspaw notes in his article on <a href="https://queue.acm.org/detail.cfm?id=2353017">fault injection in production</a>:</p>
<blockquote>
<p>[GameDay] exercises aren’t meant to discover how engineering teams handle working under time pressure with escalating and sometimes disorienting scenarios. That needs to come from the postmortems of actual incidents, not from handling faults that have been planned and designed for.</p>
</blockquote>
<p>Proactive failure testing and post-incident reviews go hand in hand. As we will see next, it’s a mistake to assume that doing enough of the former makes up for neglecting the latter (and arguably vice versa). Besides, neither of the two methods is a substitute for, say, proper monitoring and unit testing. All these practices complement each other.</p>
<h2 id="too-brittle-or-too-reliable">Too Brittle or Too Reliable</h2>
<p>It goes without saying that no amount of Chaos Engineering will fix a <a href="http://www.laputan.org/mud/">Big Ball of Mud</a> (when duct tape is holding the architecture together). Try to design for failure at all levels of your system. Address known issues <em>before</em> inviting more chaos into brittle infrastructure.</p>
<p>At the same time, don’t overdo it. Solve the business problem at hand. You’re not Google. Well, actually, Google is a bad example because they do know that <a href="https://medium.com/@jerub/embracing-risk-74bd876a64da">systems can be too reliable</a>:</p>
<blockquote>
<p>If a system gets too reliable, then the team who runs it feels like they need to keep it that reliable, even though there are potential failure modes that are very expensive to mitigate.</p>
</blockquote>
<p>Chaos Engineering must make good economic sense (remember: it’s a means to an end).</p>
<h2 id="systems-will-continue-to-fail">Systems Will Continue to Fail</h2>
<p>I’ve argued before that <a href="/premortems-the-art-of-negative-visualization">negative visualization</a> – imagining what could go wrong to prepare for disruption – is an essential part of Chaos Engineering. However, it’s also one of its major limitations, as Allspaw points out in his article mentioned above:</p>
<blockquote>
<p>The faults and failure modes are contrived. They reflect the fault designer’s imagination and therefore can’t be comprehensive enough to guarantee the system’s safety. While any increase in confidence in the system’s resiliency is positive, it’s still just that: an increase, not a completion of perfect confidence. Any complex system can (and will) fail in surprising ways, no matter how many different types of faults you inject and recover from.</p>
</blockquote>
<p>It may sound overly pessimistic, but while Chaos Engineering surely is a net plus, <a href="/impermanence-the-single-root-cause">impermanence</a> makes sure that <a href="/the-myth-of-the-root-cause">all complex systems will fail</a> no matter how hard we try to avoid it (which is exactly why postmortems are so important). The Holy Grail of Automation – introducing faults automatically instead of manually – won’t change that fact a bit. Don’t fool yourself and set realistic expectations.</p>
<p>Now I can almost hear you scream: “But there’s <a href="https://people.ucsc.edu/~palvaro/molly.pdf">Lineage-driven fault injection</a>! We don’t need to dream up any failure modes!” I hear you. LDFI is, without a doubt, a great achievement and I’m looking forward to seeing more implementations of it in the wild. However, everyone who has read the paper knows that LDFI comes with its own list of requirements and limitations, which is probably the reason why we haven’t heard much about it outside of academia so far.</p>
<h2 id="the-paradox-of-automated-fault-injection">The Paradox of Automated Fault Injection</h2>
<p>Speaking of automation, there’s an interesting paradox worth highlighting. Again, quoting from Allspaw:</p>
<blockquote>
<p>If the faults that are injected (even at random) are handled in a transparent and graceful way, then they can go unnoticed. You would think this was the goal: for failures not to matter whatsoever when they occur. This masking of failures, however, can result in the very complacency that they are intended (at least should be intended) to decrease. In other words, when you’ve got randomly generated and/or continual fault injection and recovery happening successfully, care must be taken to raise the detailed awareness that this is happening – when, how, where, etc. Otherwise, the failures themselves become another component that increases complexity in the system while still having limitations to their functionality (because they are still contrived and therefore insufficient).</p>
</blockquote>
<p>We like to talk at length about how visibility is key to detecting issues during chaos experiments – and maybe aborting experiments to prevent further harm – but let’s not forget building awareness of what’s going on in the first place.</p>
<h2 id="infinite-variability">Infinite Variability</h2>
<p>I recently found <a href="https://arxiv.org/abs/1211.1949">a paper on resilience</a> that, among other things, tries to illustrate that antifragility does not exist, at least not in the sense of systems being universally antifragile. Here’s the relevant passage:</p>
<blockquote>
<p>In general, for systems subjected to variability, noise, shocks and other random perturbations, it is possible to develop strategies that, on average, benefit from variability, but not any variability. Such strategies are designed to profit from the variability of particular stressors. Simultaneously, they are vulnerable to other stressors.</p>
</blockquote>
<p>Whether you believe in antifragility or not (<a href="/antifragility-101">I have my doubts</a>), the variability argument is spot-on. No system can withstand all turbulent conditions, but only specific ones – some of them thanks to Chaos Engineering. The somewhat worn-out vaccination analogy drives the point home: We inject something harmful into a system to build an immunity to <em>it</em>, where “it” refers to the particular “disease” we want to fight. While we might be better off overall, nobody and nothing ends up being invincible this way.</p>
<p>We can also look at the problem of variability in terms of state space, as Caitie McCaffrey <a href="https://queue.acm.org/detail.cfm?id=2889274">has done for us</a>:</p>
<blockquote>
<p>Fault-injection testing forces [failures] to occur and allows engineers to observe and measure how the system under test behaves. If failures do not occur, this does not guarantee that a system is correct since the entire state space of failures has not been exercised.</p>
</blockquote>
<h2 id="you-cant-have-a-rollback-button">You Can’t Have a Rollback Button</h2>
<p>Last but not least, let’s look at a seemingly innocuous implementation detail.</p>
<p><a href="https://blog.skyliner.io/you-cant-have-a-rollback-button-83e914f420d9">The rollback button is a lie</a>. That’s not only true for application deployments but also for fault injection, as both face the same fundamental problem: state. Yes, you might be able to revert the direct impact of non-destructive faults, which can be as simple as stopping to generate CPU/memory/disk load or deleting traffic rules created to simulate network conditions. But no, you <em>can’t</em> roll back what has been inflicted on everything else in the system – the targeted application and everything that interacts with it. A prime example is corrupt or incorrect data stored in a database/queue/cache due to a program crash.</p>
<p>One might argue that it is the very goal of chaos experiments to reveal such weaknesses, and that’s exactly right. However, having a rollback/revert button promising to quickly get back to safety is, strictly speaking, a scam. That isn’t to say we should do away with these safeguards entirely. It just means we should stop implying that all, or even most, actions can be quickly reversed, which may cause engineers to take more risk than necessary.</p>
<h2 id="wrapping-up">Wrapping Up</h2>
<blockquote>
<p>And we each forget, every day, how much care we need to take when using our seemingly benign tools; they are so useful and so sharp.<br />
– Michael Harris, The End of Absence</p>
</blockquote>
<p>Chaos Engineering has many benefits – I can’t imagine a tech world without it – but all that glitters is not gold. Please stop pretending that it is.</p>
Antifragility 1012017-09-21T00:00:00+02:00https://sharpend.io/antifragility-101<p><em>Antifragility.</em></p>
<p>If you’ve been wondering what this term means, you’re not alone. To be honest, I also had a hard time understanding the concept of antifragility, in particular how it compares to <em>resilience</em>. Fortunately, I know that writing – and the research that goes along with it – are perfect for both gaining and sharing knowledge, so I put this article together.</p>
<h2 id="resilience-by-example">Resilience by Example</h2>
<p>To understand antifragility, I think it’s helpful to understand resilience first. Here’s one definition of resilience (there are different definitions, but let’s stick with this one for antifragility to make any sense):</p>
<blockquote>
<p>A system is resilient if it can adjust its functioning prior to, during, or following events (changes, disturbances, and opportunities), and thereby sustain required operations under both expected and unexpected conditions.<br />
– <a href="http://erikhollnagel.com/ideas/resilience-engineering.html">Erik Hollnagel</a></p>
</blockquote>
<p>Resilience is something a system does, not something a system has.</p>
<p>There are many well-known patterns to build web systems that are resilient to certain kinds of failures. We use <a href="http://docs.aws.amazon.com/autoscaling/latest/userguide/">auto scaling groups</a>, for example, for clusters to maintain a minimum number of EC2 instances (server capacity) to absorb disturbances and continue serving user requests. If an auto scaling group considers an instance to be unhealthy, it will automatically terminate that instance and launch a replacement. The infrastructure can “heal itself” by recovering from (some but not all) failures. That’s one example of resilience as defined above.</p>
<p>With that out of the way, let’s take a first look at antifragility and how it’s different from resilience.</p>
<h2 id="the-antifragile-gets-better">The Antifragile Gets Better</h2>
<p>The Wikipedia page on <a href="https://en.wikipedia.org/wiki/Antifragility">Antifragility</a> is a good start if you want to learn the very basics. Here are some of the more interesting bits (emphasis mine):</p>
<ul>
<li>
<p>“Antifragility is a property of systems that increase in capability, resilience, or robustness as a result of stressors, shocks, volatility, noise, mistakes, faults, attacks, or failures. It is a concept developed by Professor Nassim Nicholas Taleb in his book, <a href="https://en.wikipedia.org/wiki/Antifragile">Antifragile</a> and in technical papers.”</p>
</li>
<li>
<p>“Antifragility is fundamentally different from the concepts of resiliency (i.e. the ability to recover from failure) and robustness (that is, the ability to resist failure).”</p>
</li>
<li>
<p>Taleb explains the differences this way: “Antifragility is beyond resilience or robustness. <strong>The resilient resists shocks and stays the same; the antifragile gets better.</strong>”</p>
</li>
</ul>
<p>In the auto scaling example above, the system is able to restore the desired capacity shortly after losing a server instance – it resists the shock and stays the same.</p>
<p>So far so good, but how, as Taleb claims, do antifragile systems get better? In other words, how do they <em>benefit</em> from disturbances?</p>
<p>One of the few practical examples of antifragility I understood intuitively is related to our body. The human body is an antifragile system because it gets better – faster and stronger – through physical training. It will adapt to the stress of exercise with increased fitness if the stress is above a certain threshold, but not too high either (a process called <a href="https://en.wikipedia.org/wiki/Adaptation">adaptation</a>).</p>
<p>While that example is relatable, I still had trouble applying the idea of antifragility to areas like software development and web operations.</p>
<h2 id="potential-downside--potential-upside">Potential Downside < Potential Upside</h2>
<p>Eager to learn more about antifragility in the context of DevOps, I read <a href="http://www.oreilly.com/webops-perf/free/antifragile-systems-and-teams.csp">Antifragile Systems and Teams</a> by Dave Zwieback. This short report turned out to be the <em>best</em> summary of the topic I’ve seen so far. I highly recommend reading it (especially if you can’t stand Taleb’s writing style).</p>
<p><strong>“The main property of antifragile systems”, Dave writes, “is that the potential downside due to stress (and its retinue) is lower than the potential upside, up to a point.”</strong></p>
<p>Some examples from the report:</p>
<ul>
<li>
<p>“Vaccination makes a population antifragile because the downside (a small number of individuals having negative side effects) is small in comparison to the upside (an entire population gaining immunity to a disease).”</p>
</li>
<li>
<p>“[With BitTorrent] the more our file is requested, the more robust to failure and available it becomes because parts of it are stored on a progressively larger number of computers. […] our cost of distributing this file would remain constant – not so for the cost of making systems more robust to anticipate higher demand or improve resiliency.”</p>
</li>
<li>
<p>“[The potential downside of frequent deployments] is smaller than the potential upside. […] customers receive higher-quality products and services (i.e., value) faster and at a lower cost than is possible with traditional, risk- and volatility-averse approaches.” (Put another way: <a href="/if-it-hurts-do-it-more-often">If it hurts, do it more often.</a>)</p>
</li>
</ul>
<p>Dave goes on to show how the layers of DevOps (culture, automation, measurement, and sharing) can contribute to the antifragility of organizations, concluding that there’s “significant overlap in practices of DevOps organizations and those that seek the benefits of antifragility”. After all, “DevOps embraces and makes use of disorder, randomness, and, ultimately, impermanence”.</p>
<h2 id="chaos-engineering">Chaos Engineering</h2>
<p>Speaking of <a href="/impermanence-the-single-root-cause">embracing impermanence</a>: if it’s possible for systems to benefit from shocks – to become more robust as a result – the idea of injecting faults <em>on purpose</em> suddenly doesn’t sound so crazy anymore, right?</p>
<p>In fact, there’s a discipline called <a href="/the-discipline-of-chaos-engineering">Chaos Engineering</a> centered around this idea. Michael Nygard, the author of <em>Release It!</em>, put it well in <a href="http://blog.cognitect.com/blog/2016/2/18/the-new-normal-from-resilient-to-antifragile#comment-2523487867">this comment</a>:</p>
<blockquote>
<p>Chaos engineering is a technique to create antifragility. That is, if you evolve toward systems that survive that kind of chaos, then your systems will exhibit antifragility.</p>
</blockquote>
<blockquote>
<p>However, one caveat: antifragility is not a universal or omnidimensional characteristic. Chaos engineering causes your system to evolve toward antifragility toward <em>those kind of stresses</em>.</p>
</blockquote>
<p>Antifragile systems might benefit from variability, but not any variability. A system can’t be universally antifragile similar to how it can’t resist any failure.</p>
<blockquote>
<p>Example: Chaos monkey kills EC2 instances. In response, you build autoscaling, masterless clusters. That helps when machines die, but not when whole regions die. Or when DNS fails. Or when data gets corrupted. Or when the marketplace changes. Etc.</p>
</blockquote>
<p>The potential downside of Chaos Engineering (occasional service interruptions) is smaller than the potential upside (better overall customer experience), up to a point (experiments causing severe damage that affect customers). While I don’t believe that web infrastructure itself can be antifragile (I might be wrong), it seems plausible to say that <strong>Chaos Engineering <em>creates</em> antifragility by enabling teams to improve their infrastructure through experimentation.</strong></p>
<hr />
<p>So, is antifragility a useful concept? I honestly still don’t know what to think of it. Writing this article led to some insightful conversations that made me question most things I <em>believe</em> to know about resilience. Among other things, I learned that antifragility might be superfluous depending on what definition of resilience you use. I therefore almost decided against publishing. However, I also realized that I’m still learning, that this piece is part of my journey. I promise it won’t be my last take on the topic.</p>
Impermanence: The Single Root Cause2017-08-25T00:00:00+02:00https://sharpend.io/impermanence-the-single-root-cause<p><img src="/assets/images/rust-wave.jpg" alt="" /></p>
<p>If you google for “impermanence”, <a href="https://en.wiktionary.org/wiki/impermanence">you will learn</a> that the word means “lack of permanence or continued duration”. This definition is admittedly very vague and very boring. If you look a bit further though, you will discover that impermanence is also the name of an essential <a href="https://en.wikipedia.org/wiki/Impermanence">doctrine of Buddhism</a>. The doctrine says that “all temporal things, whether material or mental, are compounded objects in a continuous change of condition, subject to decline and destruction”. Well, I don’t believe in Buddha, or Jesus for that matter. But I do strive to understand complex systems (<a href="/how-complex-web-systems-fail-part-1">and how they fail</a>). If religion might help me just a little bit in that regard, I’m more than willing to listen.</p>
<p>Let’s try to interpret the doctrine, starting with the core statement:</p>
<p><em>All things are compounded objects in a continuous change of condition.</em></p>
<p>Compounded objects are objects formed by combining two or more parts. In complex systems like web systems, it’s safe to assume that <em>all</em> things are compounded. Every hardware component and every piece of software consists of multiple parts.</p>
<p>Given this insight, allow me to take some mental leaps:</p>
<p><em>All systems are in a continuous change of condition.</em></p>
<p><em>All systems are changeable by nature.</em></p>
<p>And finally, to close the loop:</p>
<p><em>All systems are impermanent.</em></p>
<p><strong>In fact, constant change is a prerequisite for systems to function – to fulfill their purpose.</strong></p>
<p>Despite their catchy name, even <a href="https://martinfowler.com/bliki/ImmutableServer.html">immutable servers</a> aren’t truly immutable (sorry). For servers to do anything useful – processing HTTP requests, streaming log messages, rendering cat pictures – countless changes have to take place in both soft- and hardware. Change is indispensable.</p>
<p>On a related note, we practice <a href="/the-discipline-of-chaos-engineering">Chaos Engineering</a> to learn something new about our systems by <em>deliberately imposing change</em> on them. In a sense, we embrace the fact that all systems are impermanent.</p>
<p>But why is this understanding useful? Why should you care?</p>
<p>At this point, it’s time to admit that I first read about impermanence in Dave Zwieback’s superb book, <a href="http://shop.oreilly.com/product/0636920033981.do">Beyond Blame</a>, which led me to write the article, <a href="/learning-from-failure-and-success-through-postmortems">Learning From Failure and Success Through Postmortems</a>. He not only taught me about the meaning of the word but also, and more importantly, introduced me to this powerful idea:</p>
<p><strong>Impermanence is the single root cause for all failures and successes.</strong></p>
<p>In <em>Beyond Blame</em>, Dave explains it as follows: “The root cause for both the functioning and malfunctions in all complex systems is impermanence (i.e., the fact that all systems are changeable by nature). Knowing the root cause, we no longer seek it, and instead look for the many conditions that allowed a particular situation to manifest. We accept that not all conditions are knowable or fixable.”</p>
<p>Let that sink in for a minute. (We’ll get to the details in a bit.)</p>
<p>Eager to learn more, I also read Dave’s free report, <a href="http://www.oreilly.com/webops-perf/free/antifragile-systems-and-teams.csp">Antifragile Systems and Teams</a>, which devotes the first chapter to a brief summary of impermanence, this time going into more detail.</p>
<p>The report starts by repeating the core idea – “systems start, stop, or continue working” due to their “changeable, impermanent nature”, which is the single root cause – and goes on to explain why this theoretical understanding is indeed useful: “it reminds us that all functioning systems will eventually break down”. That knowledge, in turn, “frees us from looking for the ‘single root cause’ of outages, and from the mistaken belief that there is none”. (As Sidney Dekker famously put it: “What you call root cause is simply the place where you stop looking any further.”)</p>
<p>Having accepted impermanence, we might be tempted to blame it for each and every incident. Doing so, however, would be a mistake, depriving us of the opportunity to learn from failure. Besides, we are engineers! As Dave rightly observes, we “cannot accept that things break or function entirely randomly”. I certainly can’t. And rather than giving up our profession and going shopping, we should try hard to “identify [at least] some of the conditions” contributing to the success and failure of our systems, i.e., “conditions that we can actually impact” such as infrastructure design or collaboration in the workplace.</p>
<p>By finding and fixing those conditions – potentially <a href="/writing-your-first-postmortem">through postmortems</a> – we’re able to improve our organizations and systems in a meaningful way.</p>
<p><strong>In conclusion, we need to stop wasting our time looking for the single root cause. Impermanence is the one cause of all functioning systems and all outages. Period. We should rather <a href="/on-finding-root-causes">focus on the conditions</a> leading to both good and bad situations.</strong></p>
A Primer on Automating Chaos (Gremlin)2017-08-09T00:00:00+02:00https://sharpend.io/a-primer-on-automating-chaosLearning From Failure and Success Through Postmortems2017-07-28T00:00:00+02:00https://sharpend.io/learning-from-failure-and-success-through-postmortems<p><img src="/assets/images/yin-yang.jpg" alt="" /></p>
<p><em>Yin and yang – failure and success.</em></p>
<p>The other day, when I was listening to the <em>Beats, Rye & Types</em> podcast, I noticed this sharp statement, which I had to jot down immediately:</p>
<blockquote>
<p>With the traditional methods of dealing with failure… you can get to a certain threshold of safety – and you hit sort of a plateau beyond which you cannot go. In order for you to go beyond that, you have to start approaching safety in this new way, basically focussing on learning, removing blame, building healthy organizations.</p>
</blockquote>
<p>It was no other than Dave Zwieback, the author of <a href="http://shop.oreilly.com/product/0636920033981.do">Beyond Blame: Learning From Failure and Success</a>, who said this on <a href="http://beatsryetypes.com/episodes/2016/01/25/episode-44-blame-and-bias-with-dave-zwieback.html">the show</a>. He’s one of the best-known proponents of using blameless postmortems – or learning reviews, as he prefers to call them – to address fragility within complex systems and organizations. The interview with Dave made me want to read his book, which turned out to be an excellent decision. Despite being a short read (69 pages of content), it’s a masterpiece packed with valuable insights – some of them entirely new to me.</p>
<h2 id="learning-through-postmortems">Learning through postmortems</h2>
<p>Learning from the past is very difficult, yet it’s precisely what we need to do for our organizations to succeed. If we look a bit more closely, there’s no shortage of opportunities to learn from the past – server outages are a prime example – provided that we stop the finger-pointing and “go beyond blame and punishment”, as Dave puts it.</p>
<blockquote>
<p>With the perfect vision afforded by hindsight, we can spend a lot of time ruminating on what we could or should have done. That is counterproductive; the past is past. We need to acknowledge and learn from our mistakes, and move forward, focusing instead on what we will do now and in the future.</p>
</blockquote>
<p>Fixing the technical issues that manifest during outages is important, but not sufficient. We also need a structured process to truly learn from these events. That’s what postmortems are for. They help us understand why incidents happen and how to prepare our company – “outages are symptoms of trouble somewhere deeper in our organization” – and the systems we run for the future.</p>
<p>Indeed, I can say without exaggeration that <a href="/writing-your-first-postmortem">conducting postmortems</a> has been one of the most rewarding – though initially uncomfortable – experiences of my engineering career. Postmortem conversations can be tense when stakes are high. It’s also hard work to “mentally transport ourselves to the past” while we’re under the influence of cognitive biases (we always are). Hindsight bias, in particular, remains an obstacle to incident investigation, making it impossible to assess human performance accurately after the fact.</p>
<blockquote>
<p>“If only Mike didn’t troubleshoot the router,” for example, is not describing what actually happened, and instead of learning from the past, we’re engaging in a kind of lazy (but very comforting) wishful thinking. “Mike could have asked for help,” or “Mike should have done more testing in the lab,” or “Mike didn’t do the right thing,” are all counterfactuals, and are all evidence of hindsight bias.</p>
</blockquote>
<h2 id="all-actions-are-gambles">All actions are gambles</h2>
<p>A closely related cognitive error is outcome bias, which refers to the tendency to judge a decision by its eventual outcome (which isn’t known at decision time, of course). The same routine command we’ve used hundreds of times in the past – one that might have saved the day more than once – would suddenly crash the server because the system has drifted into failure, unnoticed by anyone. Now we’re no longer the hero; we’re just some careless cowboy administrator. That’s outcome bias at work.</p>
<p>It’s important to understand that <em>every</em> outcome, successful or not, is the result of a <em>gamble</em>. In his influential paper <a href="http://web.mit.edu/2.75/resources/random/How%20Complex%20Systems%20Fail.pdf">How Complex Systems Fail</a>, Richard Cook observes the following:</p>
<blockquote>
<p>[In complex systems] all practitioner actions are actually gambles, that is, acts that take place in the face of uncertain outcomes. The degree of uncertainty may change from moment to moment. That practitioner actions are gambles appears clear after accidents; in general, post hoc analysis regards these gambles as poor ones. But the converse: that successful outcomes are also the result of gambles; is not widely appreciated.</p>
</blockquote>
<p>Why is that so hard to comprehend? <em>Beyond Blame</em> has one answer:</p>
<blockquote>
<p>It’s a reflection of our misunderstanding of how complex systems function… Failure is a normal part of complex systems, yet it’s always so surprising when they fail. Why aren’t we more surprised when they function? … In complex systems failure is absolutely normal and expected. Malfunction is as ‘normal’ as ‘regular’ functioning.</p>
</blockquote>
<h2 id="we-get-most-thingsright">We get most things right</h2>
<p>Accepting that malfunction and “normal operation” are part of complex systems is one thing; actually learning from both failures <strong>and</strong> successes is another. I agree with Dave that postmortems tend to place too much focus on the former while neglecting the latter:</p>
<blockquote>
<p>We also want to learn not just what went wrong… but what went right – what usually goes right. We’re typically overly focused on failures, forgetting that the same systems – including the people working in them – produce both positive and negative outcomes. Mostly positive, in fact – we certainly don’t have outages every hour or even every day!</p>
</blockquote>
<p>Given the tech industry’s obsession with failure, I find this encouraging. We get <em>most</em> things right. Something to remember.</p>
<p>To wrap this up, here’s what we should do to prevent future incidents, which are inevitable, as best as we can:</p>
<blockquote>
<p>Learning from both failures and successes. Feeding these learnings as signals back into the system, which will change and adapt to this new information. That’s why air travel has become as safe as it is over time – every time there is an accident or near-accident, it’s investigated, and the results are fed back into the system. This system includes air craft, traffic control, weather, engineers, and so on.</p>
</blockquote>
<p>I couldn’t have said it better myself.</p>
<h2 id="further-reading">Further reading</h2>
<p>I’ve been studying complex systems, postmortems, root cause analysis, etc. for some time now. Feel free to look into the following articles if any of this is of interest to you:</p>
<ol>
<li><a href="/how-complex-web-systems-fail-part-1">How Complex Web Systems Fail - Part 1</a> and <a href="/how-complex-web-systems-fail-part-2">Part 2</a></li>
<li><a href="/the-myth-of-the-root-cause">The Myth of the Root Cause: How Complex Web Systems Fail</a></li>
<li><a href="/writing-your-first-postmortem">Writing Your First Postmortem</a></li>
<li><a href="/on-finding-root-causes">On Finding Root Causes</a></li>
</ol>
What I Learned From Hacking Video Games2017-07-12T00:00:00+02:00https://sharpend.io/what-i-learned-from-hacking-video-games<p>In high school and later in university I had what most people would consider a rather odd hobby: <strong>video game hacking.</strong> More precisely, I was into creating <a href="https://en.wikipedia.org/wiki/Cheating_in_video_games">cheat codes</a> for Playstation games that would, for example, make you invincible against enemy attacks or allow you to drive through walls in your favorite racing game. However, I’m not talking about the kinds of cheat codes that can be activated by entering secret passwords or pressing controller buttons in a particular sequence. I’m talking about modifying the game’s data and program code at runtime by <em>reverse engineering</em> its mechanics.</p>
<p>In the late 90s and early 2000s, you could buy off-the-shelf cheat devices for Playstation 1 (PSX) and Playstation 2 (PS2), like the <a href="https://en.wikipedia.org/wiki/Xploder">Xploder</a> and <a href="https://en.wikipedia.org/wiki/Code_Breaker">CodeBreaker</a>. Those devices worked by injecting a small program into memory between booting the console and starting a game. That program would then be running in the game’s background with full access to its memory. It was relatively easy for cheat devices to “freeze” things like energy points, ammo, money, etc. by constantly writing the same values to certain memory addresses (protections such as <a href="https://en.wikipedia.org/wiki/Address_space_layout_randomization">ASLR</a> were unheard of at the time).</p>
<figure class="">
<img src="/assets/images/xplorer.jpg" alt="" /><figcaption>
<a href="https://github.com/simias/rustation/wiki/Xplorer-FX-setup">Cheat devices like the Xploder/Xplorer could be connected to the parallel I/O port of the PSX</a>
</figcaption></figure>
<p>Best of all, the first cheat devices for PSX let you find <em>your own</em> cheat codes. This process boiled down to dumping the game’s RAM to a PC (via printer cable) where it could be searched and compared with other dumps by special “trainer” software. We did all of this on – hold your breath – Windows 98. There was a hex editor with a live view of the game’s memory; you could manipulate bytes and see the effect instantly. You could also set breakpoints, which was by far my favorite feature as it made for very sophisticated cheat codes – so-called assembly hacks that would overwrite game logic. Over the years, I’ve created thousands of codes this way.</p>
<figure class="">
<img src="/assets/images/x-link.png" alt="" /><figcaption>
<a href="http://imgur.com/8dTzclH">The powerful X-Link trainer software for PSX</a>
</figcaption></figure>
<p>When the golden age of PSX hacking was over, something bad happened: all of a sudden, companies stopped sharing their tools and started encrypting everything. It was unfortunate, but at the same time, it provided an opportunity for me to improve my limited programming and reverse engineering skills. So it happened that we wrote <a href="https://github.com/mlafeldt/ps2rd">our own software</a> for hacking PS2 games. Along the way, we cracked about anything we could get our hands on, from <a href="https://github.com/mlafeldt/cb2util">proprietary encryption schemes</a> to <a href="https://www.consolecopyworld.com/psx/psx_libcrypt_tutorial.shtml">anti-cheat protections</a>.</p>
<p>All of this was a ton of fun. As a matter of fact, it’s how I got into computers and programming in the first place. But it’s more than that, much more. Here are 10 things I learned from hacking video games, most of them only obvious in hindsight:</p>
<ol>
<li>
<p><strong>Programming.</strong> I taught myself the C programming language to write my first “proper” hacking tools. All the cool kids were using it for low-level system stuff. This was also when I became a fan of command-line tools. (These days, I dig <a href="/go-mental-models-and-side-effects">Go and Rust</a>, probably for similar reasons.)</p>
</li>
<li>
<p><strong>Bits and bytes.</strong> Video game hacking helped me understand: hexadecimal numerals, Boolean algebra, binary formats such as ELF, memory layout of executables, interrupts, breakpoints, assemblers, disassemblers, hex editors, and a ton more.</p>
</li>
<li>
<p><strong>Cryptography.</strong> Poking around the <a href="https://github.com/mlafeldt/xpcrypt">Xploder PSX code encryption</a> was my first experience in cryptography. I figured out where the device stored the unencrypted codes in RAM and did a dumb brute-force attack to break the different schemes. Since then, I’ve been fascinated by crypto.</p>
</li>
<li>
<p><strong>Networking basics.</strong> The PS2 comes with an Ethernet port/adapter, which is what we used to transfer memory dumps to a PC using a simple client-server implementation. I had to learn the basics of TCP/IP, DNS, network programming, etc. to achieve that.</p>
</li>
<li>
<p><strong>GNU/Linux.</strong> Writing code for game consoles usually involves cross-compiling under a Linux-y environment. That’s how I was introduced to gcc, bash, make, grep, and other GNU utilities – first on Windows via Cygwin and MinGW, later on Ubuntu.</p>
</li>
<li>
<p><strong>Open source.</strong> I remember how a hacker called <em>Parasyte</em> made the source code of his tools available to the public. I learned so much from his C code that I started open sourcing my programs in the same way. <a href="https://en.wikipedia.org/wiki/Homebrew_(video_games)">Homebrew</a> made sure I always had a plethora of side projects going on. I also enjoyed the social side of open source and bulletin boards.</p>
</li>
<li>
<p><strong>Version control.</strong> Subversion, Mercurial, Git. I tried them all. If I remember correctly, I was among the first who <a href="https://github.com/ps2dev">moved homebrew over to GitHub</a>. Using GitHub and other platforms also sparked my interest in writing good documentation.</p>
</li>
<li>
<p><strong>Curiosity.</strong> I believe that my strong desire to know or learn something partially stems from tinkering with video games back in the days.</p>
</li>
<li>
<p><strong>Freedom.</strong> Another <a href="/blog/what-motivates-me">important motivator</a>. I deliver my best work when I have the freedom to accomplish tasks my way. I learned this early on. But there’s also freedom in the sense that I was convinced that game hacking had to be free. No code encryption. No proprietary formats. An end to control.</p>
</li>
<li>
<p><strong>Grit.</strong> I used to spend weeks reverse engineering a video game or cheat device. Today, I still enjoy solving tricky technical problems over a long period – <a href="/perfectionism-and-programming">without losing myself in details</a>.</p>
</li>
</ol>
<p>I didn’t put this list together as a means to brag – it’s not that impressive anyway. No, I mainly wrote this for nostalgia (remembering the good old days) and introspection (trying to understand myself better).</p>
<p>While times have changed – I barely pick up a game controller anymore – I probably wouldn’t be the software engineer I am today if it wasn’t for game hacking. In fact, I may not even be a programmer at all. It still amazes me how such a geek hobby ultimately turned into a lifelong passion and career.</p>
<hr />
<p>In 2009, <a href="https://web.archive.org/web/20170918002756/http://gamehacking.org/qna/22">I did a Q&A</a> with the founder of GameHacking.org. While some of the discussed topics might not make sense to someone who has never hacked Playstation games, I think the interview still provides a bit more context. Back in the days, I used the nickname <em>misfire</em>. Naming is hard.</p>
Embracing Failure in a Container World2017-06-28T00:00:00+02:00https://sharpend.io/embracing-failure-in-a-container-world<p>What follows is the text of my presentation, <em>Embracing Failure in a Container World</em> that I gave at <a href="https://containerdays.io/">ContainerDays</a> in Hamburg this year. There’s no recording available, so I figured it would be fun to turn the presentation into an article. I edited the text slightly for readability and added some links for more context. You can find the original slides <a href="https://speakerdeck.com/mlafeldt/embracing-failure-in-a-container-world">here</a>.</p>
<hr />
<p><img src="/assets/images/2017-06-28/01.png" alt="" /></p>
<p>Hi, and welcome to my talk, “Embracing Failure in a Container World”.</p>
<p>Today I want to show you some practices and tools you can use to make <em>your</em> container systems more resilient to failures.</p>
<p><img src="/assets/images/2017-06-28/02.png" alt="" /></p>
<p>My name is Mathias.</p>
<p>I’m <code class="language-plaintext highlighter-rouge">@mlafeldt</code> on Twitter, GitHub, and pretty much anywhere else on the internet.</p>
<p><img src="/assets/images/2017-06-28/03.png" alt="" /></p>
<p>I live here in Hamburg and <a href="/blog/im-joining-gremlin-inc/">I work remotely</a> for a US startup called Gremlin Inc.</p>
<p>We obviously like to <em>break things on purpose</em>, but I will tell you more about Gremlin at the end of the talk.</p>
<p>Right now, let’s talk about a fun topic…</p>
<p><img src="/assets/images/2017-06-28/04.png" alt="" /></p>
<p>Outages.</p>
<p>Here are three of the better-known ones from this year:</p>
<ol>
<li>GitLab’s infamous “team-member-1” accidentally removed a folder on the wrong server, resulting in a <a href="https://about.gitlab.com/2017/02/10/postmortem-of-database-outage-of-january-31/">long database outage</a>.</li>
<li>We probably all noticed the <a href="https://aws.amazon.com/message/41926/">major S3 incident</a> earlier this year, which also affected other AWS services like EC2.</li>
<li>Last but not least, a <a href="https://patch.com/washington/seattle/starbucks-down-stores-across-country-affected-reported-outage">recent server outage</a> halted sales at many Starbucks stores in the US, but at least they gave out free coffee.</li>
</ol>
<p>I bet you also suffered from other outages not listed here.</p>
<p>But what’s the lesson?</p>
<p><img src="/assets/images/2017-06-28/05.png" alt="" /></p>
<p>The lesson is that sooner or later, <a href="/the-myth-of-the-root-cause">all complex systems will fail</a>.</p>
<p>There will always be something that can – and will – go wrong. No matter how hard we try, we can’t build perfect software, nor can the companies we depend on.</p>
<p>Even S3, which has had a long track record of availability, will fail in surprising ways.</p>
<p><img src="/assets/images/2017-06-28/06.png" alt="" /></p>
<p>I found this quote from <a href="https://www.slideshare.net/try_except_/large-scale-kubernetes-on-aws-at-europes-leading-online-fashion-platform-container-days-hamburg">Henning’s talk</a> to fit in nicely here. That’s why I turned it into a slide.</p>
<p>Speaking of complex systems that fail, “There’s always something with Docker in production.”</p>
<p>It’s funny because it’s true.</p>
<p><img src="/assets/images/2017-06-28/07.png" alt="" /></p>
<p>So we live in this imperfect world – things break all the time, that’s just how it is. All we can do is accept it and focus on the things we can control: creating a quality product or service that is resilient to failures.</p>
<p>Add redundancies, use auto scaling, gracefully degrade whenever possible, decrease coupling between system components – those are well-known design patterns to make systems more resilient to failures.</p>
<p><img src="/assets/images/2017-06-28/08.png" alt="" /></p>
<p>Well, at least that’s the theory. Building robust systems in practice is a lot harder, of course. How do you actually know you’re prepared for the worst in production?</p>
<p>Sure, you can learn from outages after the fact. That’s what postmortems are for. Postmortems are awesome, <a href="/writing-your-first-postmortem">I’m a big fan</a>. However, learning it the hard way shouldn’t be the only way to acquire operational knowledge.</p>
<p>Waiting for things to break in production is <em>not</em> an option.</p>
<p>But what’s the alternative?</p>
<p><img src="/assets/images/2017-06-28/09.png" alt="" /></p>
<p>The alternative is to break things on purpose. And <a href="/the-discipline-of-chaos-engineering">Chaos Engineering</a> is one particular approach to doing just that. The idea of Chaos Engineering is to be proactive – to inject failures <em>before</em> they happen in production.</p>
<p>Intentionally terminate cluster machines, delete database tables, inject network latency. Be creative. These actions help us verify that our infrastructure can cope with these failures, and to fix it otherwise.</p>
<p>However, Chaos Engineering is not only about testing assumptions, but it’s also about learning new things about our systems, like discovering hidden dependencies between components.</p>
<p>Chaos Engineering was originally formalized by Netflix. Check out their website – <a href="https://principlesofchaos.org/">principlesofchaos.org</a> – for more details.</p>
<p><img src="/assets/images/2017-06-28/10.png" alt="" /></p>
<p>Before we move on, let me give you a bit more context.</p>
<p>It’s fair to say that I learned most of the things I know about web infrastructure in my four years at <a href="https://www.jimdo.com/">Jimdo</a>, especially when I was part of the team responsible for Wonderland, which is Jimdo’s internal PaaS.</p>
<p>In fact, we gave a <a href="https://speakerdeck.com/mlafeldt/a-journey-through-wonderland">presentation about Wonderland</a> at ContainerDays last year. I want to spare you the details today. It’s enough to say that Wonderland uses ECS under the hood, which is Amazon’s cluster scheduler for Docker containers.</p>
<p>Also, Wonderland powers 100% of Jimdo’s production infrastructure. It’s not just some toy Docker project; it’s the real thing.</p>
<p><img src="/assets/images/2017-06-28/11.png" alt="" /></p>
<p>Now back to Chaos Engineering.</p>
<p>One thing we did to implement Chaos Engineering at Jimdo was to run so-called GameDay exercises on a regular basis.</p>
<p>On a typical GameDay, we would…</p>
<ol>
<li>Gather the whole team in front of a big screen.</li>
<li>Think up failure modes and estimate the impact of those failures. For example, we asked ourselves, “What would happen if we terminated 5 cluster instances at once or if this critical microservice went down?”</li>
<li>Go through all chaos experiments together, breaking things on purpose just as we planned.</li>
<li>Write down the measured impact.</li>
<li>Create follow-up tickets for all flaws uncovered this way.</li>
</ol>
<p>I can assure you that we found a number of issues every single time, many caused by missing timeouts and unexpected dependencies, and also some bugs in open source software we were using.</p>
<p>To put it in a nutshell, GameDays are great. They helped us improve Wonderland to the point where we had to test PagerDuty during GameDays to find out if our monitoring was still working – because <a href="/complacency-the-enemy-of-resilience">alerts were so rare</a>.</p>
<p><img src="/assets/images/2017-06-28/12.png" alt="" /></p>
<p>One particular tool we used during GameDays was <a href="/chaos-monkey-for-fun-and-profit">Chaos Monkey</a>.</p>
<p><em>Who of you has used Chaos Monkey before? And in production? (Asking the audience. For the record, only Jimdo employees raised their hands twice.)</em></p>
<p>Chaos Monkey was built by Netflix to terminate EC2 instances randomly during business hours. The goal is for the infrastructure to survive terminations without any customer impact.</p>
<p>I think you all agree that it’s better to test this proactively in the office than at 4 am in the night when shit really hits the fan.</p>
<p><img src="/assets/images/2017-06-28/13.png" alt="" /></p>
<p>Unfortunately, the configuration of Chaos Monkey is a bit complex. It’s a Java program that has dozens and dozens of settings. That was one problem, the other one was deployment.</p>
<p>Since we already had this amazing container platform at Jimdo, I decided to dockerize Chaos Monkey and solve the configuration issue at the same time.</p>
<p><a href="https://github.com/mlafeldt/docker-simianarmy">This Docker image</a> is the result. We ended up using it to run one monkey per environment (one in staging, one in production), which gave us a solid foundation for running chaos experiments on GameDays.</p>
<p><img src="/assets/images/2017-06-28/14.png" alt="" /></p>
<p>To give you an example, this shows the basic usage of the Docker image.</p>
<p>We pass the required AWS credentials to the image and instruct it to consider all EC2 auto scaling groups of the corresponding AWS account for termination.</p>
<p>When running, Chaos Monkey will randomly pick one auto scaling group and terminate one instance of that group. That’s how it works.</p>
<p><img src="/assets/images/2017-06-28/15.png" alt="" /></p>
<p>Chaos Monkey usually runs continuously, killing an instance every hour or so. However, we only needed it during GameDays or whenever we had to do some resilience testing.</p>
<p>Luckily, Chaos Monkey also comes with a REST API that allows us to terminate instances on demand. And I wrote a <a href="https://github.com/mlafeldt/chaosmonkey">command-line tool</a> in Go to make use of that API.</p>
<p><img src="/assets/images/2017-06-28/16.png" alt="" /></p>
<p>Again, here’s an example.</p>
<p>We first install the <code class="language-plaintext highlighter-rouge">chaosmonkey</code> command-line tool using <code class="language-plaintext highlighter-rouge">go get</code>.</p>
<p>We then tell it where Chaos Monkey is running, what auto scaling group we want to target, and how the instance should be terminated (Chaos Monkey supports different chaos strategies, with “shutdown” being the default).</p>
<p>We can also be fancy and terminate multiple instances in a row, which is helpful to find out when a cluster loses quorum, for example.</p>
<p>The tool can do a couple more things not shown here. I recommend checking out the <a href="https://github.com/mlafeldt/chaosmonkey#readme">README on GitHub</a> for more information.</p>
<p><img src="/assets/images/2017-06-28/17.png" alt="" /></p>
<p>One last tip on Chaos Monkey:</p>
<p>It’s possible to <a href="https://github.com/mlafeldt/docker-simianarmy/blob/master/docs/notifications.md">hook it up to Slack</a> so that your team is aware of ongoing chaos experiments. This slide shows an example message from one of our Slack channels.</p>
<p>For obvious reasons, visibility is important when doing Chaos Engineering.</p>
<p><img src="/assets/images/2017-06-28/18.png" alt="" /></p>
<p>So far, we talked about Chaos Monkey and how it can be used to terminate instances. Of course, terminating hosts is only one way to inject failures into your system. There’s certainly a lot you can learn from this exercise, but there are also many more places where your infrastructure can fail.</p>
<p>We are at ContainerDays, so I guess we’re interested in impacting not only the hosts where our containers are running on but also the containers themselves, right?</p>
<p><img src="/assets/images/2017-06-28/19.png" alt="" /></p>
<p>This is where <a href="https://github.com/gaia-adm/pumba">Pumba</a> comes in.</p>
<p>Pumba is an open source tool designed for injecting failures into running Docker containers. Just as Chaos Monkey, you can use it to simulate real-world events. For example, you can kill specific containers or inject network errors into them.</p>
<p><img src="/assets/images/2017-06-28/20.png" alt="" /></p>
<p>Here’s an example showing Pumba in action.</p>
<p>We first start a test Ubuntu container. We then instruct Pumba to delay all outgoing network traffic from that container for 60 seconds. That’s basically how you can simulate latency to external services used by an application running in that container.</p>
<p><img src="/assets/images/2017-06-28/21.png" alt="" /></p>
<p>Now you might wonder how this works under the hood.</p>
<p>Internally, Pumba uses a tool called <code class="language-plaintext highlighter-rouge">tc</code> that talks to the traffic shaping API of the Linux kernel. However, it would be impractical to bake <code class="language-plaintext highlighter-rouge">tc</code> into every application container you want to attack.</p>
<p>So what Pumba does is spawn a sidecar container, which has <code class="language-plaintext highlighter-rouge">tc</code> preinstalled. And it will spawn that sidecar container in the network namespace of the target Ubuntu container. That’s what the <code class="language-plaintext highlighter-rouge">--net</code> option does here.</p>
<p>When done, Pumba will do the same thing to remove the traffic rule from the kernel again.</p>
<p>Pretty magical, but works well in practice.</p>
<p><img src="/assets/images/2017-06-28/22.png" alt="" /></p>
<p>At the beginning of the talk, I promised you to tell you more about Gremlin Inc.</p>
<p>So what do we do?</p>
<p><img src="/assets/images/2017-06-28/23.png" alt="" /></p>
<p><a href="https://www.gremlin.com/">Gremlin</a>, our product, provides failure as a service. It’s another powerful tool for resilience testing.</p>
<p>You can use Gremlin to attack hosts and containers. In the future, it will also be possible to attack applications directly through application-level fault injection.</p>
<p>We support a variety of attacks that you can run from our Web UI or the command line. One unique feature of Gremlin is that it can safely revert all impact. We also provide security features like auditing and access control out of the box.</p>
<p>Gremlin is currently in closed beta. If you want to give it a try, talk to me, and I will send you an invite.</p>
<p><img src="/assets/images/2017-06-28/24.png" alt="" /></p>
<p>Let’s wrap this up. What are the takeaways from this talk?</p>
<p>The main takeaway is that building resilient systems requires experience with failure. Don’t wait for things to break in production, but rather inject failure proactively in a controlled way.</p>
<p>Use one or more of the many chaos tools available today. Use Chaos Monkey, use Pumba, use Gremlin – whatever works for you.</p>
<p>Please keep in mind that it’s important to start small! Don’t wreak havoc on production from day one and tell your boss it was my idea. Instead, start by experimenting with a virtual machine or a staging environment. Then slowly ramp up your testing efforts.</p>
<p><img src="/assets/images/2017-06-28/25.png" alt="" /></p>
<p>For those of you who have enjoyed this presentation and want to learn more about Chaos Engineering or SRE in general, I wrote a lot of articles on these topics in the last year. Check out my <a href="/production-ready">Production Ready mailing list</a> and feel free to talk to me about anything afterward.</p>
<p><img src="/assets/images/2017-06-28/26.png" alt="" /></p>
<p>Thank you.</p>
Go, Mental Models, and Side Effects2017-06-14T00:00:00+02:00https://sharpend.io/go-mental-models-and-side-effects<p><a href="/every-day-we-must-sweep">I recently wrote about my struggle</a> of learning Rust and Java at the same time. Since then, a lot of people have asked me if I finally came to grips with both programming languages. The short answer: yes and no. Yes, I do enjoy building things in Rust, although its compile times still demand <a href="https://xkcd.com/303/">considerable patience</a>. And no, if I were given a choice, I wouldn’t use Java for my daily work, mainly because it feels heavy and tends to encourage programmers to write overly verbose code with too much abstraction, which goes against <a href="/simplicity-a-prerequisite-for-reliability">one of my core values</a>.</p>
<p>One thing I’ve learned over the years – sometimes the hard way – is that learning a new language usually doesn’t happen in a vacuum. More often than not, there already exists a body of code, infrastructure, conventions, team values, etc. you need to grasp if you want to contribute in a meaningful way. This has led me to believe that the degree to which one enjoys a programming language invariably depends on the given environment. In other words, if you don’t like language X, it would be too easy – and certainly unfair – to put the sole blame on language X. Look around; X is only part of the equation.</p>
<p>That caveat aside, however, I still believe that some programming languages make it easier to write <em>maintainable code</em> than others. One excellent example is <a href="https://golang.org/">Go</a>.</p>
<h2 id="the-virtues-of-go">The virtues of Go</h2>
<p>Go is a straightforward, no-frills programming language that is almost <em>non-magical</em>. You might even call Go boring, but <a href="/sometimes-boring-is-better">being boring is actually a good thing</a> when it comes to technology. After all, software should behave predictably and accomplish its goals without too many surprises. <strong>Code readability – and maintainability – first, language features second.</strong> What good is the latter without the former?</p>
<p>Go’s tooling is mature (okay, maybe except for dependency management). There’s a vast amount of <em>working</em> libraries, making it a perfect fit for developing delightful command-line tools, lightning-fast web servers, and robust distributed systems. Betting on Go, which is a breeze to learn, makes it easier for businesses to hire and onboard new developers (in particular if onboarding only involves telling the new employee where the source code can be found, but I digress).</p>
<p>Having used Go successfully in production for years, I could go on and on and on, but I’ll stop here.</p>
<p>Of course, Go is not a panacea. You can still write terrible code in it. More than once did I end up in interface hell when trying to navigate some Go code. <a href="http://peter.bourgon.org/blog/2017/06/09/theory-of-modern-go.html">Peter Bourgon also rightly notes</a> that despite being a non-magical language, there are still a few ways magic can creep in through the use of global state – beware of unpredictable side effects!</p>
<p>Yet I’d argue that, by and large, Go’s explicitness and lack of fancy language features make it harder for programmers to create a <a href="http://www.laputan.org/mud/">Big Ball of Mud</a>. Go is relatively easy to read, understand, and reason about. In other words, <strong>Go minimizes the effort required to build a mental model of a program</strong>.</p>
<p><img src="/assets/images/mental-models.jpg" alt="" /></p>
<p>What’s a mental model? I’m glad you’re asking.</p>
<h2 id="mental-models-101">Mental models 101</h2>
<blockquote>
<p>[A mental model] is a representation of the surrounding world, the relationships between its various parts and a person’s intuitive perception about his or her own acts and their consequences. Mental models can help shape behavior and set an approach to solving problems and doing tasks.</p>
</blockquote>
<p>That’s what <a href="https://en.wikipedia.org/wiki/Mental_model">Wikipedia</a> says. Here’s another explanation, this one <a href="http://queue.acm.org/detail.cfm?id=3068754">specific to programming</a>:</p>
<blockquote>
<p>At a fundamental level, all software describes changes in the state of a system over time. Because the number states and state transitions in software can have combinatorial complexity, programmers necessarily rely on approximations of system behavior (called mental models) during development. The intent of a mental model is to allow programmers to reason accurately about the behavior of a system.</p>
</blockquote>
<p>We all carry different, more or less accurate images of how something works in our heads, be it the weather or cars or code.</p>
<p>This goes for all engineering teams. Alice might be the one who knows the most about the new distributed cron solution (written in Go, of course). Hence she possesses the most comprehensive model of this particular component and how it interacts with other components. However, when it comes to service deployments and secrets management, Bob and Ted might have a clearer picture of what’s going on.</p>
<h2 id="systems-blindness">Systems blindness</h2>
<p><a href="/systems-blindness-and-how-we-deal-with-it">Systems are invisible to our eyes</a>. The best thing we can do is understand them indirectly through mental models, and then perform actions based on these models. Unfortunately, these models tend to be incomplete – or just plain wrong. Blame complexity: the more complex a system, the more difficult it becomes for our brain to build a correct mental model of it.</p>
<p>Remember that I warned you about global state in Go? Well, in actuality, <strong>there are no side effects, just effects that result from our flawed understanding of the system.</strong></p>
<p><a href="http://queue.acm.org/detail.cfm?id=3068754">Bugs occur</a> because of an incomplete mental model:</p>
<blockquote>
<p>Since mental models are approximations, they are sometimes incorrect, leading to aberrant behavior in software when it is developed on top of faulty assumptions…. The most sinister bugs occur when programmers falsely believe their mental models to be complete…. Throwing away the mental model is crucial to forming a sound hypothesis [when debugging].</p>
</blockquote>
<p>Good programming languages can <em>reduce the scope</em> of the mental model developers must maintain, affecting both programming and debugging in a positive way. I believe Go meets the criteria. I miss using it on a daily basis.</p>
Getting Things Right With Checklists2017-06-01T00:00:00+02:00https://sharpend.io/getting-things-right-with-checklists<p><img src="/assets/images/checklist.jpg" alt="" /></p>
<p>I’ve always been a very organized person, sometimes to an obsessive degree (<em>don’t touch my stuff!</em>). I like being on top of things. I don’t want to miss any steps when carrying out a task. Naturally, checklists have long been one of my favorite tools to get things done.</p>
<p>When it comes to software development, for example, it’s hard to beat <a href="https://help.github.com/articles/about-task-lists/">GitHub’s task lists</a>. To plan and prioritize personal tasks, I also like to write checklists on paper. And of course, there are plenty apps out there to choose from, all based on the simple but powerful idea of the checklist.</p>
<p>But although I’ve used different checklists for many years and firmly believe in their effectiveness, I realized that my knowledge of them was superficial at best. That’s why I decided to read <a href="http://atulgawande.com/book/the-checklist-manifesto/">The Checklist Manifesto</a> by Atul Gawande, which was recommended to me multiple times.</p>
<p>The book is about Gawande’s journey in designing a checklist for surgery. He, quite surprisingly, draws most lessons from aviation and construction, where checklists are standard practice, and applies them to medicine. He ultimately succeeds (spoiler alert). The final 19-item <a href="http://www.who.int/patientsafety/safesurgery/checklist/">Surgical Safety Checklist</a> has gone on to show a significant reduction in the number of complications and deaths by decreasing errors and increasing teamwork in surgery.</p>
<p>The following is a summary of what I learned from reading <em>The Checklist Manifesto</em>.</p>
<h2 id="man-is-fallible">Man is fallible</h2>
<p>The first question that comes to mind is why we need checklists in the first place?</p>
<p>The short answer: we humans need checklists because we are <em>fallible</em>. More precisely, we fail at what we set out to do for three reasons:</p>
<ul>
<li><strong>Necessary fallibility.</strong> Even enhanced by technology, some things are simply beyond our capacity and will remain outside our understanding and control.</li>
<li><strong>Ignorance.</strong> We make mistakes because we don’t know enough about the world and how it works. We still can’t predict the weather accurately. There are still diseases we cannot cure.</li>
<li><strong>Ineptitude.</strong> We fail to apply existing knowledge correctly. Despite our best efforts, we write buggy code. We construct buildings that collapse.</li>
</ul>
<p>In today’s complex world, our main problem is ineptitude. Gawande writes:</p>
<blockquote>
<p>science has filled in enough knowledge to make ineptitude as much our struggle as ignorance. […] Know-how and sophistication have increased remarkably across almost all our realms of endeavor, and as a result so has our struggle to deliver on them.</p>
</blockquote>
<p>He continues to depict in great detail how medicine in particular “has become the art of managing extreme complexity”:</p>
<blockquote>
<p>To save this one child, scores of people had to carry out thousands of steps correctly: placing the heart-pump tubing into her without letting in air bubbles; maintaining the sterility of her lines, her open chest, the exposed fluid in her brain; keeping a temperamental battery of machines up and running. The degree of difficulty in any one of these steps is substantial. Then you must add the difficulties of orchestrating them in the right sequence, with nothing dropped, leaving some room for improvisation, but not too much.</p>
</blockquote>
<p>Getting things right is becoming harder and harder every day – even for specialists who have received intense training. We need a different strategy.</p>
<p><strong>Checklists.</strong></p>
<p>Checklists do not only compensate for the limits of our memory and attention, but they also lead to higher performance by forcing us to be disciplined:</p>
<blockquote>
<p>people can lull themselves into skipping steps even when they remember them. In complex processes, after all, certain steps don’t always matter. […] Checklists seem to provide protection against such failures. They remind us of the minimum necessary steps and make them explicit. They not only offer the possibility of verification but also instill a kind of discipline of higher performance.</p>
</blockquote>
<h2 id="teamwork">Teamwork</h2>
<p>So checklists help us apply the knowledge we have consistently and correctly. However, that by itself is not enough. Gawande continues:</p>
<blockquote>
<p>the volume and complexity of what we know has exceeded our individual ability to deliver its benefits correctly, safely, or reliably. Knowledge has both saved us and burdened us.</p>
</blockquote>
<p>No one can do everything anymore, neither in aviation, nor in construction, nor in any other complex environment. The Genius Master Builder is dead. Besides, not everything can be reduced to a simple recipe.</p>
<p>But who says that a checklist only contains, say, tasks for constructing a building? In addition to decreasing errors, checklists can also be used to increase teamwork and communication in order to overcome individual weaknesses and deal with unexpected problems as a team. These <em>communication checklists</em> are used in construction and elsewhere to force specialists to talk to each other:</p>
<blockquote>
<p>While no one could anticipate all the problems, [experts] could foresee where and when they might occur. The checklist therefore detailed who had to talk to whom, by which date, and about what aspect of construction – who had to share (or “submit”) particular kinds of information before the next steps could proceed.</p>
</blockquote>
<p>Above all, the goal of checklists is to <strong>embrace a culture of teamwork and discipline</strong>. <a href="https://www.ted.com/talks/atul_gawande_how_do_we_heal_medicine">Complexity requires group success</a>.</p>
<h2 id="what-makes-a-good-checklist">What makes a good checklist?</h2>
<p>Now that we know the “why” of checklists, let’s take a look at the “how”. What exactly makes a good checklist?</p>
<p>According to the book, good checklists are:</p>
<ul>
<li>Precise, efficient, to the point</li>
<li>Reminders of the <em>most important steps</em>, not comprehensive how-to guides</li>
<li>Between five and nine items long (the limit of working memory)</li>
<li>Quick and simple tools to support the skills of experts</li>
<li>Practical – tested in the real world</li>
<li>Easy to use even in difficult situations</li>
<li>Written in simple and exact language, using familiar terms of the profession</li>
<li>Frequently revisited and refined to help rather than hinder</li>
</ul>
<p>Unless the moment is obvious, we also need to define <em>pause points</em> at which a checklist is supposed to be used in a process. We can choose between two options: a DO-CONFIRM checklist (perform tasks from memory, then stop to verify) or a READ-DO checklist (carry out tasks as you check them off). Both have pros and cons.</p>
<h2 id="adoption">Adoption</h2>
<p>To reap the benefits of checklists, we need to be willing to adopt them as part of our daily work and, ultimately, our company culture. Checklists alone cannot make anyone follow them.</p>
<p>As Gawande knows all too well, we will meet resistance when introducing a checklist at a larger scale. Besides <a href="/the-pros-and-cons-of-eating-your-own-dog-food">eating your own dog food</a>, he came to the conclusion that the first people using a new checklist should “have the seniority and patience to make the necessary modifications and not dismiss the whole enterprise”. Sounds reasonable.</p>
<p>Some people may still object that checklists are merely about ticking boxes, as this passage from the book points out well:</p>
<blockquote>
<p>It somehow feels beneath us to use a checklist, an embarrassment. It runs counter to deeply held beliefs about how the truly great among us – those we aspire to be – handle situations of high stakes and complexity. The truly great are daring. They improvise. They do not have protocols and checklists. Maybe our idea of heroism needs updating.</p>
</blockquote>
<p>It’s important to understand that a checklist isn’t just a protocol you’re supposed to follow mindlessly. It is rather <em>supporting</em> us in our work:</p>
<blockquote>
<p>The checklist gets the dumb stuff out of the way, the routines your brain shouldn’t have to occupy itself with (Are the elevator controls set? Did the patient get her antibiotics on time? Did the managers sell all their shares? Is everyone on the same page here?), and lets it rise above to focus on the hard stuff (Where should we land?).</p>
</blockquote>
<p>Even with checklists, there’s still plenty of room for individual judgment and performance.</p>
<h2 id="a-production-readiness-checklist">A production-readiness checklist</h2>
<p>Of course, I wouldn’t write this article if there wasn’t a direct relationship to software development and running systems in production. <strong>Web systems are also among the complex environments where we can – and should – use checklists for greater efficiency, consistency, and safety.</strong></p>
<p>I want to wrap this up with a practical example. While <a href="https://gitlab.com/gitlab-com/runbooks">runbooks</a> make for great checklists in web operations, I found a different example by reading yet another book.</p>
<p>In <a href="http://shop.oreilly.com/product/0636920053675.do">Production-Ready Microservices</a>, Susan J. Fowler provides a useful checklist to decide whether a microservice is ready for production or not. According to Fowler, a production-ready service must be:</p>
<ol>
<li>Stable and reliable</li>
<li>Scalable and performant</li>
<li>Fault tolerant and prepared for any catastrophe</li>
<li>Properly monitored</li>
<li>Documented and understood</li>
</ol>
<p>For each item, the production-readiness checklist defines specific criteria that must be met. For example, this is what it takes for a microservice to be fault tolerant (3):</p>
<ul>
<li>It has no single point of failure.</li>
<li>All failure scenarios and possible catastrophes have been identified.</li>
<li>It is tested for resiliency through code testing, load testing, and chaos testing.</li>
<li>Failure detection and remediation has been automated.</li>
<li>There are standardized incident and outage procedures in place within the microservice development team and across the organization.</li>
</ul>
<p>I think it’s a fantastic idea to create production-readiness checklists for system components if your goal is to build standardized systems across an engineering organization. For my part, I’m planning to go deeper into the topic and come up with some checklists of my own.</p>
<p><strong>Our jobs aren’t too complicated to reduce to a checklist. In fact, we are more likely to fail if we don’t try.</strong></p>
Premortems: The Art of Negative Visualization2017-05-17T00:00:00+02:00https://sharpend.io/premortems-the-art-of-negative-visualization<p><img src="/assets/images/rome.jpg" alt="" /></p>
<p>I’ve become <a href="https://www.instagram.com/p/BTwWzXGD2_E/">a big fan</a> of Ryan Holiday’s work. It comes as no surprise that one of his books, <em>Ego Is The Enemy</em>, has inspired me to write <a href="/every-day-we-must-sweep">Every Day We Must Sweep</a> a couple of weeks ago. I couldn’t help it and also read the critically acclaimed predecessor, <em><a href="http://theobstacleistheway.com/">The Obstacle Is The Way</a></em>, shortly after. It too turned out to be a remarkable book, drawing on the vast experience of Emperor Marcus Aurelius and other famous historical figures. Again, I highlighted dozens and dozens of passages I considered to be noteworthy. One particular chapter stood out to me, though. It’s called “Anticipation (Thinking Negatively)” and it has a direct, almost disturbing relevance to my daily work. If it were possible, I would give you a copy of it right now, but I’m afraid my interpretation and lots of quotes will have to do for the time being.</p>
<p>The chapter begins with a discussion of <em>premortems</em>, which are said to be popular in the business world. The basic idea of a premortem is that managers encourage employees to <em>think negatively</em> – in terms of worst-case scenarios – when preparing for major events like a product launch. Holiday writes:</p>
<blockquote>
<p>In [a premortem], we look to envision what could go wrong, what will go wrong, in advance, before we start. Far too many ambitious undertakings fail for preventable reasons. Far too many people don’t have a backup plan because they refuse to consider that something might not go exactly as they wish.</p>
</blockquote>
<p>Astute readers will notice the play on words here. Premortem is, of course, the opposite of <em>postmortem</em>. With a postmortem, we’re examining something after it happened so that we can learn and improve for the next time a similar situation occurs. In the tech world, <a href="/writing-your-first-postmortem">postmortems are the ultimate tool</a> to learn from server outages and other failures.</p>
<p>Being a software engineer myself, I had to smile when reading about “premortems” for the first time. At that moment, however, I also realized that this seemingly odd term resembles what I do for a living like few other words – but read on.</p>
<p>According to Holiday, this practice – this form of <em>negative visualization</em> – can be attributed to ancient Stoic philosophers:</p>
<blockquote>
<p>A writer like Seneca would begin by reviewing or rehearsing his plans, say, to take a trip. And then he would go over, in his head (or in writing), the things that could go wrong or prevent it from happening: a storm could arise, the captain could fall ill, the ship could be attacked by pirates.</p>
</blockquote>
<blockquote>
<p>Always prepared for disruption, always working that disruption into our plans. Fitted, as they say, for defeat or victory. And let’s be honest, a pleasant surprise is a lot better than an unpleasant one.</p>
</blockquote>
<p>Rehearsing plans.</p>
<p>Imagining what could go wrong.</p>
<p>Preparing for disruption.</p>
<p>Given my background in web operations and <a href="/blog/im-joining-gremlin-inc">my current job</a>, guess what immediately came to my mind? Call me crazy, but it was the idea of <a href="/breaking-things-on-purpose">breaking things on purpose</a> – imagining and simulating potential errors in advance – in order to build systems that are resilient to failures.</p>
<p>Holiday continues:</p>
<blockquote>
<p>Your world is ruled by external factors. Promises aren’t kept. […] We are dependent on other people. […] The only guarantee, ever, is that things will go wrong. The only thing we can use to mitigate this is anticipation.</p>
</blockquote>
<p><a href="/the-myth-of-the-root-cause">All complex systems will fail</a>. There will always be something that can – and <em>will</em> – go wrong. From self-inflicted outages caused by bad configuration or buggy images to events outside our control like denial-of-service attacks or network failures.</p>
<p><em>Anticipation</em> is key: hoping for the best, preparing for the worst. (I don’t want to sound overly pessimistic, but this maxim must exist for a reason.)</p>
<blockquote>
<p>As a result of our anticipation, we understand the range of potential outcomes and know that they are not all good (they rarely are). We can accommodate ourselves to any of them. We understand that it could possibly all go wrong. And now we can get back to the task at hand. […] <strong>We are prepared for failure and ready for success.</strong></p>
</blockquote>
<p>By being well prepared, we will not be caught by surprise and won’t be disappointed. And even if there’s nothing we can do about an outage or similar events, we could use that as a practice to <em>manage our expectations</em>. Because sometimes the only way out is through.</p>
<p>Turns out the Stoics knew more about web technology than you’d think.</p>
The Discipline of Chaos Engineering (Gremlin)2017-05-03T00:00:00+02:00https://sharpend.io/the-discipline-of-chaos-engineeringThere's nothing like a good spike2017-04-19T00:00:00+02:00https://sharpend.io/theres-nothing-like-a-good-spike<p><img src="/assets/images/spikes.jpg" alt="" /></p>
<p>I’m a big proponent of <a href="http://wiki.c2.com/?SpikeSolution">spike solutions</a>. A spike is a simple end-to-end solution to a technical problem. Spikes are <em>cheap</em> – and often dirty – implementations that are meant to be thrown away after exploration. It’s fine to create multiple spikes to explore different directions when dealing with a tough engineering challenge. Spikes should only be concerned with the problem at hand, independent of existing code, best practices, and similar ceremony. That means, for the first time in your career, you’re encouraged to write some lousy shell scripts to get the job done. I do it all the time – no one will blame you.</p>
<p>Spikes are all about <em>reducing risk</em> by reducing the number of unknowns. Spikes help you verify that you’re on the right track, that what you’ve imagined is indeed possible given the constraints you face. Spikes also make for more accurate estimates of development costs. Having trouble predicting how long a feature is going to take? Or if it’s feasible at all? Create a spike, and your decisions will be all the wiser.</p>
<p>The great thing about spikes is that they can prove you wrong <em>before</em> you’ve wasted a lot of resources trying to build The Real Thing™. They help you <a href="/perfectionism-and-programming">overcome perfectionism</a> by making sure you don’t lose yourself in details right from the get-go. No wonder spikes are listed among the <a href="http://www.extremeprogramming.org/rules.html">Rules of Extreme Programming</a>.</p>
<p>That’s enough praising for now. Let me give you a real-world example. Last year, when I was still working at Jimdo, we were looking for a more reliable service for running periodic batch jobs inside our PaaS. One of the candidates that caught my attention was <a href="https://www.nomadproject.io/">Nomad</a>, the cluster scheduler developed by HashiCorp.</p>
<p>As the <a href="http://shop.oreilly.com/product/0636920030355.do">Lean Enterprise</a> book has taught us, we should only spend time on automation for products or features once they have been validated. Anything else is <em>wasteful</em>. Rather than rushing to automate Nomad’s setup in AWS on day one (which later turned out to take a couple of weeks), I decided to create a spike first. Before adding yet another tool to our stack and taking on <a href="/the-burden-of-running-systems">the burden of operating it</a>, I wanted to learn more about Nomad’s capabilities and figure out if it was the right choice for us.</p>
<p>As I mentioned, we were looking for a <em>reliable</em> cron solution. The spike’s purpose was to convince us that Nomad was at least worth a closer look. To that end, I asked myself: what would be the simplest setup to achieve that goal?</p>
<p>I ended up doing some <a href="https://gist.github.com/mlafeldt/d74f2d5c3594216ca5683629b8af855c">local testing with Vagrant</a> based on a demo that ships with Nomad. For the spike, I started a Nomad server and two Nomad clients inside the virtual machine managed by Vagrant. I then created a minimal periodic job that would send a message to a Slack channel every minute. Based on this quick experiment, I was able to learn more about Nomad’s mechanics and test its resilience to different kinds of injected failures, like killing one or both clients. (During testing, I actually encountered a serious bug in Nomad causing successful batch jobs to be run again after restarting a stopped client. Luckily, that bug had been fixed in master the day before…)</p>
<p>All in all, the spike was a success. We continued doing more experiments in AWS, gradually going from a one node setup to operating Nomad in a highly-available fashion.</p>
<p>In a sense, spikes are low-risk experiments – not unlike <a href="/breaking-things-on-purpose">Chaos Engineering experiments</a> – to validate assumptions early on. I consider them an essential part of my software development toolbox.</p>
<p>Next time you hit a roadblock, a spike or two might be all you need to move forward.</p>
Every Day We Must Sweep2017-04-05T00:00:00+02:00https://sharpend.io/every-day-we-must-sweep<p>I’m in the midst of learning two new programming languages: Java and Rust.</p>
<p>I’m struggling, and it’s my fault.</p>
<p>At Gremlin, <a href="/blog/im-joining-gremlin-inc">the company I joined a month ago</a>, we use Java for our API service in the backend and Rust for our client, which injects infrastructure failures, and also Rust for our daemon, which communicates with the API.</p>
<p>It’s been a while since I felt this stupid when looking into a new programming language, let alone two that are so radically different.</p>
<p>To be honest, I’m not completely new to either Java or Rust. I wrote some Java Card programs in my first job after university (Java Card is a small Java runtime for running crypto code on SIM cards). That was almost ten years ago. Since then, I’ve made my fair share of jokes about Java, covering topics like software bloat, enterprise readiness, and – of course – desktop updates. Recently, it all came back to bite me when it literally took me days to get started with Eclipse, Gradle, dependency management, and whatnot.</p>
<p>I also have an on-off relationship with Rust, the systems programming language praised for being super fast and super safe. Before Gremlin, I successfully rewrote two of my C tools in Rust. However, I never <em>really</em> learned the language and its myriad of programming concepts. Yes, Rust lets you control almost everything, but as a beginner, it’s also hard to compile <em>anything</em>. As a result, Rust continues to make me look bad. (By the way, compile times are ridiculously slow compared to Go, which I’ve used for the last four years.)</p>
<p><img src="/assets/images/rust-makes-me-look-bad.jpg" alt="" /></p>
<p>So apparently, I don’t particularly like Java and Rust, and this is just another rant about programming languages? Actually, no, not at all.</p>
<h2 id="ego-is-the-enemy">Ego is the enemy</h2>
<p>Some days ago, I finished reading <a href="http://egoistheenemy.com/">Ego Is The Enemy</a> by Ryan Holiday. As the title says, the book is devoted to the treacherous nature of ego. It draws on a vast array of stories of people who didn’t let ego control their actions and decisions on their road to success. On the other hand, the book contains just as many tales of individuals who lost the inner battle against ego.</p>
<p>Here are a few of the quotes I highlighted on my Kindle:</p>
<blockquote>
<p>If ego is the voice that tells us we’re better than we really are, we can say ego inhibits true success by preventing a direct and honest connection to the world around us.</p>
</blockquote>
<blockquote>
<p>As success arrives, like it does for a team that has just won a championship, ego begins to toy with our minds and weaken the will that made us win in the first place.</p>
</blockquote>
<blockquote>
<p>We must […] continue working on what got us here. Because that’s the only thing that will keep us here.</p>
</blockquote>
<blockquote>
<p>Just because you did something once, doesn’t mean you’ll be able to do it successfully forever. Reversals and regressions are as much a part of the cycle of life as anything else.</p>
</blockquote>
<blockquote>
<p>The problem is that when we get our identity tied up in our work, we worry that any kind of failure will then say something bad about us as a person.</p>
</blockquote>
<blockquote>
<p>Daniele Bolelli once gave me a helpful metaphor. He explained that training was like sweeping the floor. Just because we’ve done it once, doesn’t mean the floor is clean forever. Every day the dust comes back. Every day we must sweep.</p>
</blockquote>
<p>What in the world has this to do with me learning Java and Rust? Everything.</p>
<h2 id="being-a-beginner-again">Being a beginner again</h2>
<p>Starting a new gig is a huge decision, in part because you have to learn so many new things and relearn stuff you had long forgotten. Sure, no one can take away your hard-earned experience. However, there are other areas, like programming languages, where you have to start over, where you aren’t an expert (yet).</p>
<p>Being a beginner again is hard. I know it all too well. It can be frustrating. I’m used to producing quality work in a short time span, but projects that took hours now take days. I keep asking my (patient and helpful) coworkers about the most basic things, which often makes me feel outright dumb. Am I too hard on myself? Probably.</p>
<p>Reading Holiday’s brilliant book helped me find the true cause of my struggle: ego. It is my ego that keeps telling me that I’m better than this, that my past performance is a guarantee of future success, that I’m too professional to be playing in the amateur league again.</p>
<p>The truth is I’m all for learning new things. As an engineer, I want to be able to contribute to Gremlin’s codebase. I want to write Java and Rust code that is used in production. And yes, I do seek the <em>uncomfortable</em>. In fact, that’s one of the reasons why I quit my last job.</p>
<p>To that end, I don’t want ego to get in my way. I know I need to discard any preconceived notions about programming languages I’ve never actually used for an extended period (and which are only a means to an end anyway).</p>
<p>Despite the little experience I have at this point, I must admit that Java 8 streams are pretty neat. I like the refactoring and debugging capabilities of modern-day Java IDEs. Rust reminded me of <a href="/what-i-learned-from-hacking-video-games">my game hacking days</a> more than any other language. Cargo, Rust’s package manager, is terrific. I’m sure there are many more amazing things to learn – but only if I manage to leave my ego at the door and continue working on what got me here in the first place.</p>
<p><strong>Every day the dust comes back. Every day we must sweep.</strong></p>
Breaking Things on Purpose (Gremlin)2017-03-16T00:00:00+01:00https://sharpend.io/breaking-things-on-purposeSometimes Boring Is Better2017-03-08T00:00:00+01:00https://sharpend.io/sometimes-boring-is-better<p><img src="/assets/images/boring.jpg" alt="" /></p>
<p>The recent announcement of <em>Docker Enterprise Edition</em> brought back some bad memories. After having used Docker in production for two years, I honestly don’t have a lot of faith in its overall stability and direction – enterprise or not. CoreOS failed to ship a stable Docker version on multiple occasions despite their efforts to provide a production-grade <em>Container Linux</em>. Throw <em>ecs-agent</em> into the mix, and you’re guaranteed to have a lot of headaches due to broken/blocked cluster updates. Once bitten, twice shy.</p>
<p>To be fair, using a supposedly stable tool or OS does not relieve you from doing your own sanity testing. As engineers know, many problems only manifest themselves at scale in production (which is why <a href="/from-zero-to-staging-and-back">verifying things in staging</a> only works to a certain degree). Also, remember that we’re dealing with complex web systems, which are <a href="/how-complex-web-systems-fail-part-1">prone to failure</a>.</p>
<p>So yes, we should cut Docker, CoreOS, and Amazon ECS some slack. All of them have their place. All of them are by themselves groundbreaking technologies that, taken together, enable us to do amazing things, like running a company-wide PaaS for production services. No, I don’t blame them. In fact, I’m glad they exist. On the other hand, they’re still good examples for making the following point.</p>
<p>What do Docker, CoreOS, and ECS have in common? All three are relatively <em>new technologies</em>. Some might even call them “bleeding edge” (I won’t). In any case, all three are the opposite of boring – they’re rather hip and shiny. The point of this article is that, <strong>when it comes to technology, sometimes boring is actually better</strong>.</p>
<p>Over the last couple of months, I’ve read a number of articles on the merits of choosing boring technology. <a href="http://mcfunley.com/choose-boring-technology">Dan McKinley’s article</a> is without a doubt one of the best pieces on the topic. It’s worth reading from beginning to end, but here are some of my favorite quotes:</p>
<blockquote>
<p>The nice thing about boringness (so constrained) is that the capabilities of these things are well understood. But more importantly, their failure modes are well understood. […] But for shiny new technology the magnitude of unknown unknowns is significantly larger, and this is important.</p>
</blockquote>
<p>In other words, <strong>software that has been around for a decade is well understood and has fewer unknowns.</strong> Fewer unknowns mean less operational overhead, which is a good thing.</p>
<blockquote>
<p>One of the most worthwhile exercises I recommend here is to consider how you would solve your immediate problem without adding anything new. […] It’s helpful to write down exactly what it is about the current stack that makes solving the problem prohibitively expensive and difficult.</p>
</blockquote>
<p><a href="/the-burden-of-running-systems">New systems mean new problems</a>, so we should think twice before adding anything new to an otherwise boring and well-understood stack.</p>
<blockquote>
<p>set clear expectations about migrating old functionality to the new system. The policy should typically be “we’re committed to migrating,” with a proposed timeline. The intention of this step is to keep wreckage at manageable levels, and to avoid proliferating locally-optimal solutions.</p>
</blockquote>
<p>Timeboxing migrations is an excellent idea I probably should have applied a couple of times in the past. As for locally-optimal solutions, like using new technology X in a single place without good reason, John Allspaw had the following to say <a href="https://medium.com/s-c-a-l-e/microservices-monoliths-and-laser-nail-guns-how-etsy-finds-the-right-focus-in-a-sea-of-cf718a92dc90">in an interview about Etsy</a>:</p>
<blockquote>
<p><strong>we want to exploit the advantages of having a relatively finite number of well-known tools.</strong> […] the advantages of being more optimal do not outweigh the advantages of using the same language [PHP] a lot. […] In the same way, there’s a massive advantage in using a default data store, MySQL.</p>
</blockquote>
<p>He points out that if something breaks for some reason, each new tool is another thing an engineer has to understand, making it more difficult for the company to be resilient:</p>
<blockquote>
<p>The thing is, when you pull something shiny and new off the shelf, there can be operational overhead. If it breaks and you’re the only one who knows how it works, then it probably wasn’t a great technical choice. […] <strong>We want to plan for a world where stuff breaks all the time. And we want to make it so that when things break they matter a lot less, that they’re not critical.</strong> That they break and we can fix them and we can adapt and be resilient.</p>
</blockquote>
<p>I, too, believe that being a bit more conservative and slowing down the pace would benefit our industry.</p>
<p>Skyliner, the AWS launch platform, is certainly a prime example of <a href="https://blog.skyliner.io/the-happy-genius-of-my-household-2f76efba535a">this philosophy</a>:</p>
<blockquote>
<p>Skyliner doesn’t use registries, scheduling, service discovery, virtualized networking, or any other advanced features. Instead, we use AWS services with proven reliability like S3, Autoscaling, and Elastic Load Balancing – services which have seen almost a decade of continuous use and improvement.</p>
</blockquote>
<blockquote>
<p>As software developers, we understand the allure of shiny new technologies, but ultimately we decided that <strong>we prefer the quiet satisfaction of sleeping through a night while on call</strong>. After all, we’re not just building a platform for our customers – we run our applications on Skyliner, too.</p>
</blockquote>
<p>So, should we give up and stop using advanced container technologies altogether? Absolutely not. What we need, first and foremost, is a <a href="https://medium.com/@bob_48171/an-ode-to-boring-creating-open-and-stable-container-world-4a7a39971443">simple, boring container implementation</a> that just works. More generally speaking, what we need are <em>stable building blocks</em>. Everything on top – our production systems – will flourish from there.</p>
<p>Docker and friends aren’t boring yet, but eventually they will be. I’m looking forward to that day.</p>
The Pros and Cons of Eating Your Own Dog Food2017-02-22T00:00:00+01:00https://sharpend.io/the-pros-and-cons-of-eating-your-own-dog-food<p>In the software industry, eating your own dog food – or <em>dogfooding</em> – is a common approach for companies to test their product or service by letting employees use it in real-life scenarios. The idea is that by using your own software just as a customer would, you can proactively validate and incrementally improve it before releasing it to the world. Besides quality control, relying on your own product also makes for good marketing.</p>
<p>One of my favorite books, <a href="https://scottberkun.com/yearwithoutpants/">The Year Without Pants</a> by Scott Berkun, contains this memorable account of dogfooding, which I found to be worth quoting in length (I added line breaks for readability):</p>
<blockquote>
<p>It turned out the adjustment was easy. The Internet Explorer team at Microsoft had an equivalent, called the daily build, where we released a version of the software every day, but it was available exclusively inside the company. Each day all the changes from the previous day were compiled and released, and everyone was expected to install and use them. <strong>This gave us regular feedback on the quality of what we were making, including nuggets of joy, or moments of misery, when new features were added.</strong></p>
</blockquote>
<blockquote>
<p>On good days, the builds were high quality, and we called those releases self-host, as in “safe to host on your computer.” Builds that were mediocre were called self-test, suggesting you install it only on a test computer (or a coworker’s when the person wasn’t looking). The worst builds were called self-toast, meaning you’d destroy whatever machine you had dared installed it on. Whenever we had three days in a row with self-toast builds, all new work stopped until we got the build quality up to a good level (a measure to prevent the project from digging a dangerously deep quality hole for itself).</p>
</blockquote>
<blockquote>
<p>Shipping on WordPress.com was the same philosophy, just accelerated and made public to customers. I didn’t find the lack of bigger plans or schedules a problem. In fact, it was mostly liberating.</p>
</blockquote>
<p>It’s true that dogfooding lends itself more to certain products than others. It doesn’t take a lot of imagination to see, for example, how it can be used effectively in these cases:</p>
<ul>
<li>Using Internet Explorer, a browser, to build Internet Explorer</li>
<li>Using Basecamp, a project management tool, to build Basecamp</li>
<li>Using GitHub, a code hosting/collaboration platform, to build GitHub</li>
</ul>
<p>In each example, the product in question is a tool engineers would need for everyday software development anyway – a perfect fit for eating your own dog food. For organizations/teams who don’t build user-facing products similar to the ones listed above, dogfooding might be limited to more constrained test scenarios, or it might be impossible to implement for the system as a whole.</p>
<p>I’ve spent the last 18 months of my career working on a project that’s suitable for dogfooding like no other before it. Building and operating <a href="/blog/a-journey-through-wonderland/">Wonderland</a>, Jimdo’s in-house PaaS, has provided us with the unique opportunity to run most of the platform’s services on the platform itself – basically everything that can be deployed via Docker.</p>
<p>We don’t throw our work over to QA and wait for bug reports (there’s no QA department anyway). More often than not, we are the first to feel the joy and, more importantly, <a href="/if-it-hurts-do-it-more-often">feel the pain</a> of using our PaaS in production. Dogfooding gives us invaluable feedback on both released features and work in progress; it helps us to detect many issues before our users do.</p>
<p>To some degree, dogfooding has also been useful for anticipating wishes and evaluating ideas expressed by other teams at Jimdo (our customers). However, it’s always a challenge when we don’t have a real need for a specific feature. For example, back when we added the ability to run periodic jobs in Wonderland, our team didn’t have a use case for them at first, so we created a somewhat contrived cron that posted some stats in a Slack channel. That’s <em>not</em> ideal.</p>
<p>While we’re the first users of our PaaS, we’re by no means the largest. That means we might not be the first team running into scaling issues. That’s unfortunate, but it’s also a matter of costs. I’m confident that, for instance, load tests in production would come in useful here. Speaking of testing, dogfooding is <em>no</em> substitute for traditional automated and usability testing. Nor is it an excuse for <a href="/the-burden-of-running-systems">depending on internal systems only</a>.</p>
<p>There’s also the danger that developers may have knowledge to make software work that a normal user will lack. In fact, it’s common for SaaS products to expose tons of system internals when being used in “developer mode” by its creators. That’s certainly useful but also something to be aware of.</p>
<p>Despite these disadvantages, I’m convinced that <strong>those who develop software should ideally be the first ones to use it on a day-to-day basis.</strong></p>
<p>Try hard to make your software a major part of your workflow. Put yourself in the shoes of existing or potential users as often as you can. Feed results back into the product’s design and code. Rinse, repeat. This way, overall quality and usability are very likely to increase over time.</p>
Perfectionism and Programming2017-02-08T00:00:00+01:00https://sharpend.io/perfectionism-and-programming<p><img src="/assets/images/perfectionism.png" alt="" /></p>
<p>I consider myself a perfectionist.</p>
<p>When I’m writing a program, I want my code to be correct. It has to get the job done. At the same time, I strive to produce code that is perfectly formatted, fast, and (to me) beautiful. That in itself wouldn’t be a problem if all I did was to follow the <a href="http://wiki.c2.com/?MakeItWorkMakeItRightMakeItFast">wisdom of Kent Beck</a> to “first make it work, then make it right, and, finally, make it fast.” Kent is right, of course. His advice is entirely reasonable. However, and here is the catch, it only works if you <em>know when to stop.</em></p>
<p>There’s nothing wrong with <a href="/always-leave-the-campground-cleaner-than-you-found-it">refactoring</a> and optimizing code per se – in fact, it’s essential to programming. But trying to Make It Right™ at any cost can be dangerous. I know this first hand. After spending countless nights in front of my computer, hacking on <a href="https://github.com/mlafeldt">open source projects</a>, I know what it’s like to get lost in implementing <em>yet another tweak</em> – a minor change that is supposed to make a difference. I know how it feels like to rewrite the same piece of code over and over again, with no end in sight. It’s frustrating, to say the least.</p>
<h2 id="writing">Writing</h2>
<p>I don’t strive for the best in everything I do, but writing is another creative endeavor where perfectionism used to get in my way.</p>
<blockquote>
<p>The biggest problem I face [with publishing content] is that I’m a perfectionist. I have a hard time writing shitty first drafts and postponing editing until after getting something down on paper first. Instead, I often try to get it “right” the first time, thereby making the writing process unnecessarily painful. The number of blog posts I’ve published this year is evidence enough of my struggle.</p>
</blockquote>
<p><a href="/blog/write-every-day">I wrote this in September 2014</a>. These days I still suck at drafting. Writing continues to be a painful process from time to time. While I’ve figured out how to publish articles on a regular basis – the not-so-secret secret is writing consistently – I’m fully aware of the fact that perfectionism is a formidable obstacle to getting things done.</p>
<p>I like the way Anne Lamott put it in her great book, <em>Bird by Bird</em>:</p>
<blockquote>
<p>Perfectionism is the voice of the oppressor, the enemy of the people. It will keep you cramped and insane your whole life, and it is the main obstacle between you and a shitty first draft. […] Perfectionism is a mean, frozen form of idealism</p>
</blockquote>
<h2 id="good-enough">Good enough</h2>
<p>Back to perfectionism and programming. Before you open your favorite editor and attempt to achieve an <a href="https://en.wikipedia.org/wiki/Perfectionism_(psychology)">unattainable ideal</a>, consider the following:</p>
<p>Unless you’re building low-latency trading systems or similar software where every microsecond counts, you probably don’t need to write the most efficient code ever conceived. Don’t fall prey to <a href="http://wiki.c2.com/?PrematureOptimization">premature optimization</a>. And unless you have a strong reason to believe that your current design is inadequate, chances are that another layer of perfectly crafted abstraction won’t save the day. Moreover, there is no need to thoroughly document every method of your code unless you’re providing an API that is actually used by somebody.</p>
<p>When I work on some piece of code and I feel the urge for perfection creeping in my head, I like to ask myself these questions:</p>
<ul>
<li>Does this change <em>really</em> make a difference? Is it worth my time? Consider engineering costs vs. value created. Think long- and short-term.</li>
<li>Does it provide value to the users of my software? Users typically don’t care about internals like program code.</li>
<li>Does it matter to my coworkers? My boss? My future self?</li>
</ul>
<p>By all means, I <em>don’t</em> want you to ship crappy code. But I do want you to remember that you have a choice – and more often than not, settling for <em>good enough</em> is a valid choice to make in a tech world that values shipping above all else.</p>
<p>At this point, I feel obliged to remind you that <a href="/the-power-of-less-code">the best code is no code at all</a>. Perfectionist or not, always start by looking for solutions that don’t involve writing code.</p>
<h2 id="tools-and-practices">Tools and practices</h2>
<p>If you’re reading this, you’re probably a programmer yourself. I guess you’re looking for some advice that’s a bit more concrete. I hear you. Here are six things that help me overcome perfectionism in programming:</p>
<ul>
<li>
<p><strong>Tooling.</strong> Remember that I want my code to be perfectly formatted? For me, tools like <a href="https://blog.golang.org/go-fmt-your-code">gofmt</a> are a godsend. Never do I have to worry about whitespace or other formatting issues again. Instead, I can turn my attention to more interesting tasks.</p>
</li>
<li>
<p><strong>Pair programming.</strong> I’ve experienced that the desire to finish work increases when working in pairs, in particular, if the work seems to be too hard or too boring or both. (Pair programming <a href="http://wiki.c2.com/?PairProgrammingBenefits">offers much more benefits</a>.)</p>
</li>
<li>
<p><strong>Code reviews.</strong> Besides pair programming, requesting a code review and getting an “LGTM” from my colleagues is another helpful indicator that my changes are good enough.</p>
</li>
<li>
<p><strong>Testing.</strong> For me, perfectionism is closely related to confidence, the confidence in the code I write. <a href="/implementing-semantic-monitoring">Passing tests</a> certainly increase the level of confidence in my work, provided I know the tests themselves are valuable.</p>
</li>
<li>
<p><strong>Spikes.</strong> Rather than losing myself in details by trying to Make It Right™ from the get-go, I prefer creating a spike first. A spike is a simple (and dirty) end-to-end solution to a given problem, which is meant to be thrown away after exploration.</p>
</li>
<li>
<p><strong>Deadlines.</strong> Having a due date, ideally imposed by someone else, helps me overcome perfectionism and procrastination. Going back to writing for a second: to keep on schedule, <em>I had to</em> press the button for publishing the very post you’re reading.</p>
</li>
</ul>
<p>Reading over this list again, it becomes apparent that it’s not only about fighting perfectionism, it’s also a set of <strong>established practices for producing quality software</strong>. I call this a win-win.</p>
<p><em>This piece was inspired by the spot-on post, <a href="https://web.archive.org/web/20171123161254/http://www.terramilitia.com/post/3/terrors-of-perfectionism/">Terrors of perfectionism</a>. Among other things, it helped me realize how often we consider a solution to a problem to be flawed and therefore temporary. Then we end up using it in production until doomsday.</em></p>
Implementing Semantic Monitoring2017-01-25T00:00:00+01:00https://sharpend.io/implementing-semantic-monitoring<p><img src="/assets/images/red-roof.jpg" alt="" /></p>
<p>2016 was an exciting year. I don’t think there was another period in the almost ten years I’ve been in the software industry where I learned so much about web infrastructure. Building and running <a href="https://speakerdeck.com/mlafeldt/a-journey-through-wonderland">Jimdo’s PaaS</a> has taught me a thing or two about reliability, scalability, usability, and other software “-ilities”. Today I want to write about one particular monitoring/testing strategy that has been invaluable to the success of our PaaS.</p>
<h2 id="the-difficulty-of-monitoring-microservices">The difficulty of monitoring microservices</h2>
<p>Our platform is composed of two dozen microservices sitting on top of Amazon Web Services (rumor has it we’re one of the largest users of ECS). When deploying a service via our CLI tool, the API request first hits AWS API Gateway, which will forward it to the corresponding microservice – our Deployer API in this case. The Deployer then enqueues a new job that will be picked up by a worker, which in turn deploys the service with the support of other microservices.</p>
<p>Breaking a system up into smaller services has a lot of benefits, but it also makes it more complex to monitor that the system as a whole is working correctly. For example, just because the services themselves report to be healthy doesn’t necessarily mean the integration points between them are fine too.</p>
<p>To verify that the Deployer works as expected, we execute a number of unit and integration tests as part of the CI pipeline. Besides, all of our services are fronted by a load balancer and backend replicas are replaced automatically at runtime when they become unhealthy. If that doesn’t help, PagerDuty will send us a friendly alert.</p>
<p>So we run some tests before deploying a new version of a microservice into production, and we use health checks to ping the service while it’s doing its job, e.g. processing API requests from impatient users. That’s a common way to validate production services. Unfortunately, it’s also a <em>missed opportunity</em>.</p>
<p>There are several problems with this approach:</p>
<ul>
<li>We run the service-specific test suite only once and stop using it altogether when the service goes into production.</li>
<li>Apart from one-off integration tests, we don’t test the interaction between microservices continually.</li>
<li>Compared to CI tests, health check endpoints are usually dumb, often merely indicating if a service is running at all.</li>
<li>Additional low-level metrics like CPU utilization or response time are useful to pinpoint the cause of trouble, but they won’t give us a holistic view either.</li>
</ul>
<p>There’s obviously a lot of room for improvement here. This is where <a href="https://www.thoughtworks.com/de/radar/techniques/semantic-monitoring">semantic monitoring</a> comes in.</p>
<p><strong>Semantic monitoring combines test execution and realtime monitoring to <em>continuously</em> verify the behavior of applications. It lends itself in particular to validating microservices and how they interact at runtime.</strong></p>
<h2 id="implementing-semantic-monitoring">Implementing semantic monitoring</h2>
<p>How to implement semantic monitoring? The short answer: by feeding the results of end-to-end tests (consumer-driven contracts) into your existing monitoring solution. Those tests typically mimic user actions via fake events, e.g. a synthetic deployment, to ensure that the system behaves semantically (hence the name, semantic monitoring). What follows is a more or less detailed summary of the implementation we’re using at Jimdo.</p>
<p>At the heart of our setup is a set of black box tests. Those tests, which are written in Go, communicate directly with our API, just as users would do. Among other things, we have tests to ensure services and <a href="/the-burden-of-running-systems">periodic jobs</a> can be deployed and deleted successfully, both in staging and in production.</p>
<p>Due to the distributed nature of such systems, most tests boil down to “X should do A, B, C within T minutes”. As one API request can create a chain of downstream calls and events that are handled asynchronously, it’s a good idea to pass along a correlation ID. We output the unique resource IDs created during testing to be able to trace events through our systems should something go wrong.</p>
<p>To execute those tests continuously, we’ve configured a Jenkins job called “System-Tests”, which will run every hour and notify us about any failures in Slack. This, in fact, used to be the whole story for a long time: a couple of black box tests executed by Jenkins and Slack notifications that were easy to miss. That is, until <a href="https://prometheus.io/">Prometheus</a> entered the picture.</p>
<p>At some point last year, we committed to using Prometheus for monitoring all the things from cluster instances to microservices running on our PaaS. And with Prometheus came the <a href="https://github.com/prometheus/pushgateway">Pushgateway</a>, which allows ephemeral and batch jobs to expose metrics in an easy way.</p>
<p><a href="https://gist.github.com/mlafeldt/7fab53f47dd73a8c34f8b95ec444ab79">We created a generic Docker image</a> to push a “freshness” metric to the Pushgateway. This metric – a timestamp plus some labels – can be used to determine whether a job was executed within a certain period or not.</p>
<p>We then added it to the System-Tests Jenkins job to write a freshness metric before and after running the tests:</p>
<figure class="highlight"><pre><code class="language-sh" data-lang="sh">push_build_metrics<span class="o">()</span> <span class="o">{</span>
docker run <span class="nt">-t</span> <span class="nt">--rm</span> <span class="nt">-e</span> <span class="nv">PUSHGATEWAY</span><span class="o">=</span><span class="s2">"</span><span class="nv">$PUSHGATEWAY_ADDR</span><span class="s2">"</span> <span class="se">\</span>
quay.io/jimdo/freshness <span class="nv">job</span><span class="o">=</span>Jenkins <span class="se">\</span>
<span class="nv">name</span><span class="o">=</span>System-Tests <span class="nv">branch</span><span class="o">=</span><span class="nv">$BRANCH</span> <span class="nv">state</span><span class="o">=</span><span class="nv">$1</span>
<span class="o">}</span>
push_build_metrics started
make <span class="nb">test
</span>push_build_metrics success</code></pre></figure>
<p>Afterward, we configured Prometheus to alert us if the job failed for three times in a row (at this point, we don’t trust Jenkins enough for this check to trigger PagerDuty at night):</p>
<figure class="highlight"><pre><code class="language-sh" data-lang="sh">ALERT SystemTestsFailing
IF <span class="nb">time</span><span class="o">()</span> - freshness<span class="o">{</span><span class="nv">name</span><span class="o">=</span><span class="s2">"System-Tests"</span>,branch<span class="o">=</span><span class="s2">"master"</span>,state<span class="o">=</span><span class="s2">"success"</span><span class="o">}</span> <span class="o">></span> 3.5<span class="k">*</span>60<span class="k">*</span>60
LABELS <span class="o">{</span>
wonderland_env <span class="o">=</span> <span class="s2">"prod"</span>
<span class="o">}</span>
ANNOTATIONS <span class="o">{</span>
summary <span class="o">=</span> <span class="s2">"System-Tests Jenkins job wasn't successful for 3 hourly runs in a row."</span>
<span class="o">}</span></code></pre></figure>
<p>And that’s the story of how we ended up implementing semantic monitoring on the cheap, based on building blocks already in place. All of the mentioned components for testing and monitoring can be used on their own. But by combining them, we can merge two separate but important verification techniques to monitor not only our microservices but also the integration points between them.</p>
<p>(If you want to learn more about monitoring microservices, I highly recommend reading <a href="http://samnewman.io/books/building_microservices/">Building Microservices</a> by Sam Newman.)</p>
Using Chaos Monkey whenever you feel like it2017-01-11T00:00:00+01:00https://sharpend.io/using-chaos-monkey-whenever-you-feel-like-it<p>The main idea of <a href="/chaos-engineering-101">Chaos Engineering</a>, we recall, is to trigger failures proactively in a controlled way to gain confidence that our production systems can withstand those failures. Chaos Engineering enables us to verify that our systems behave as we expect – and to fix them if they don’t.</p>
<p><a href="/chaos-monkey-for-fun-and-profit">In a previous article</a>, I showed you how to use Chaos Monkey for automating your first chaos experiment. For this purpose, I’ve created a customizable <a href="https://github.com/mlafeldt/docker-simianarmy">Docker image</a> of the Simian Army (which Chaos Monkey is part of) as a solid foundation for running experiments.</p>
<p>Netflix originally designed Chaos Monkey to terminate EC2 instances randomly during business hours. To that end, the tool comes with a good deal of configuration settings to control frequency, probability, type of terminations, and a lot more.</p>
<p>However, you <em>don’t</em> need to automate experiments to run continuously in order to benefit from Chaos Engineering (but doing so <em>may</em> further increase confidence in your systems). At Jimdo, we don’t use Chaos Monkey in a conventional way. In fact, the service is idle most of the time, waiting for instructions from us.</p>
<h2 id="on-demand-termination">On-demand termination</h2>
<p>Many people don’t know that, in addition to scheduled instance terminations, Chaos Monkey also supports killing instances <em>on demand</em> via its built-in <a href="https://github.com/Netflix/SimianArmy/wiki/REST">REST API</a>. So instead of inflicting chaos on your servers at random, e.g. once an hour between 10am and 5pm, it’s on you to decide if and when the monkey will perform a destructive action, without having to follow any imposed schedule.</p>
<p>Want to give it a try? With the mentioned Docker image, setup is a breeze. This single command will start a Chaos Monkey that is ready to take API requests on port 8080 (don’t worry about scheduled terminations – they’re deactivated in this example):</p>
<figure class="highlight"><pre><code class="language-sh" data-lang="sh">docker run <span class="nt">-it</span> <span class="nt">--rm</span> <span class="nt">-p</span> 8080:8080 <span class="se">\</span>
<span class="nt">-e</span> <span class="nv">SIMIANARMY_CLIENT_AWS_ACCOUNTKEY</span><span class="o">=</span><span class="nv">$AWS_ACCESS_KEY_ID</span> <span class="se">\</span>
<span class="nt">-e</span> <span class="nv">SIMIANARMY_CLIENT_AWS_SECRETKEY</span><span class="o">=</span><span class="nv">$AWS_SECRET_ACCESS_KEY</span> <span class="se">\</span>
<span class="nt">-e</span> <span class="nv">SIMIANARMY_CLIENT_AWS_REGION</span><span class="o">=</span><span class="nv">$AWS_REGION</span> <span class="se">\</span>
<span class="nt">-e</span> <span class="nv">SIMIANARMY_CHAOS_LEASHED</span><span class="o">=</span><span class="nb">false</span> <span class="se">\</span>
<span class="nt">-e</span> <span class="nv">SIMIANARMY_CHAOS_ASG_ENABLED</span><span class="o">=</span><span class="nb">false</span> <span class="se">\</span>
<span class="nt">-e</span> <span class="nv">SIMIANARMY_CHAOS_TERMINATEONDEMAND_ENABLED</span><span class="o">=</span><span class="nb">true</span> <span class="se">\</span>
mlafeldt/simianarmy</code></pre></figure>
<p>Afterward, you can talk to the API to trigger and retrieve instance terminations, or “chaos events”, as they are called here. For example, this will give you a list of past chaos events:</p>
<figure class="highlight"><pre><code class="language-sh" data-lang="sh">curl http://<span class="nv">$DOCKER_HOST_IP</span>:8080/simianarmy/api/v1/chaos</code></pre></figure>
<p>Triggering failures via the API is a bit more involved, and I won’t go into the details here. Instead, I’d like to promote a handy command-line tool I’ve written for that purpose.</p>
<h2 id="cli-goodness">CLI goodness</h2>
<p>Say hello to the <strong><a href="https://github.com/mlafeldt/chaosmonkey">chaosmonkey CLI tool</a></strong>. (Not to be confused with the same-named binary that comes with <a href="https://github.com/Netflix/chaosmonkey">Chaos Monkey v2</a>. I had the idea first, I swear.)</p>
<p>Originally developed for controlled failure injection during <a href="/chaos-engineering-101">GameDays at Jimdo</a>, the tool could also be described as “Chaos Monkey whenever you feel like it”.</p>
<p>Let’s use it to send some commands to the Chaos Monkey we just started with <code class="language-plaintext highlighter-rouge">docker run</code>. First, and most important, here’s how to trigger a new chaos event. This will block all network access to a random instance of the given EC2 auto scaling group:</p>
<figure class="highlight"><pre><code class="language-sh" data-lang="sh">chaosmonkey <span class="nt">-endpoint</span> http://<span class="nv">$DOCKER_HOST_IP</span>:8080 <span class="se">\</span>
<span class="nt">-group</span> ExampleAutoScalingGroup <span class="se">\</span>
<span class="nt">-strategy</span> BlockAllNetworkTraffic</code></pre></figure>
<p>Sometimes it’s also convenient to terminate multiple instances of an auto scaling group, e.g. to test under what conditions a cluster loses quorum. In this example, we’re going to shut down three cluster instances at intervals of 30 seconds:</p>
<figure class="highlight"><pre><code class="language-sh" data-lang="sh">chaosmonkey <span class="nt">-endpoint</span> http://<span class="nv">$DOCKER_HOST_IP</span>:8080 <span class="se">\</span>
<span class="nt">-group</span> ExampleAutoScalingGroup <span class="se">\</span>
<span class="nt">-strategy</span> ShutdownInstance <span class="se">\</span>
<span class="nt">-count</span> 3 <span class="nt">-interval</span> 30s</code></pre></figure>
<p>It’s also straightforward to list past chaos events as Chaos Monkey keeps track of everything:</p>
<figure class="highlight"><pre><code class="language-sh" data-lang="sh">chaosmonkey <span class="nt">-endpoint</span> http://<span class="nv">$DOCKER_HOST_IP</span>:8080</code></pre></figure>
<p>There are a couple more features not shown here for brevity. The AWS integration, for instance, allows you to list the auto scaling groups for a given AWS account and to wipe Chaos Monkey’s state if you want to start over. I encourage you to <a href="https://github.com/mlafeldt/chaosmonkey#readme">read the documentation</a> for further details, including installation instructions.</p>
<h2 id="chaos-monkey-at-jimdo">Chaos Monkey at Jimdo</h2>
<p>Now that I told you about the REST API and the CLI tool, I want to share how we deploy and run Chaos Monkey at Jimdo. Here are the facts:</p>
<ul>
<li>Chaos Monkey is just another service running on <a href="/blog/a-journey-through-wonderland/">Jimdo’s PaaS</a></li>
<li>For deployment, we use the <a href="https://github.com/mlafeldt/docker-simianarmy">Docker image</a> mentioned above</li>
<li>We run one monkey in production and one in <a href="/from-zero-to-staging-and-back">our staging environment</a></li>
<li>We use <code class="language-plaintext highlighter-rouge">:8080/simianarmy/</code> for the HTTP health check</li>
<li>A Nginx auth proxy protects the API endpoint (which is public on our platform)</li>
<li>For high availability, we deploy two service replicas behind an ELB (this works because we only use the REST API, no scheduled terminations)</li>
<li>We get <a href="https://github.com/mlafeldt/docker-simianarmy/blob/master/docs/notifications.md">Slack notifications</a> for all terminations</li>
<li>The <code class="language-plaintext highlighter-rouge">chaosmonkey</code> tool is installed on our bastion hosts, preconfigured and ready to use for chaos experiments</li>
</ul>
<p>That’s about it.</p>
<p>At this point, you may wonder if we really need all this complexity to kill some EC2 instances once in a while? Maybe not. It depends on your infrastructure and the type of chaos testing you’re doing. For us, this setup makes sense given the PaaS we have in place and the kind of GameDay exercises we’ve been performing on a regular basis. At the same time, I won’t recommend replacing Chaos Monkey and its many off-the-shelf features with a shell script. That’s just not sustainable in the long run.</p>
<p><strong>Resilience testing should be second nature to engineers. It’s something we should be doing more often – without fear – and open source tools like Chaos Monkey facilitate this goal. I like the idea of <a href="/bring-your-tools-with-you">having it at my disposal</a> whenever I need it.</strong></p>
The Burden of Running Systems2016-11-30T00:00:00+01:00https://sharpend.io/the-burden-of-running-systems<p><img src="/assets/images/burden.jpg" alt="" /></p>
<p>In <a href="/the-power-of-less-code">The Power of Less Code</a> I wrote:</p>
<blockquote>
<p>A couple of weeks ago, we finished the migration from Dkron to Nomad for running all periodic batch jobs of our PaaS. […] And yet there’s a much better solution we would have preferred: not [operating Nomad] in the first place, but rather outsource the task of running periodic jobs to a hosted service provider with a paid support plan.</p>
</blockquote>
<p><a href="https://medium.com/@hakibenita/very-important-idea-and-well-written-thanks-2b56e9093b2b">One reader replied</a> that the solution to not writing code wasn’t to load it off to someone else, e.g. to a SaaS vendor, as dependencies would be just as bad.</p>
<p>That’s a good point. While I still believe in every single word I wrote, this comment made me realize that my original statements lacked depth and deserve more explanation from my side.</p>
<h2 id="youre-not-paid-to-write-code">You’re not paid to write code</h2>
<p>In his excellent post, <a href="http://bravenewgeek.com/you-are-not-paid-to-write-code/">You’re not paid to write code</a>, Tyler Treat takes the same line as I do: writing code should always be the last resort and never the first option to add value to the business. However, Tyler looks at the topic from a different angle, drawing from systems theory. Some of his points are worth quoting here (emphasis mine):</p>
<blockquote>
<p>[John] Gall’s Fundamental Theorem of Systems is that <strong>new systems mean new problems</strong>. I think the same can safely be said of code – more code, more problems. Do it without a new system if you can.</p>
</blockquote>
<blockquote>
<p>Every time you write code or <strong>introduce third-party services</strong>, you are introducing the possibility of failure into your system.</p>
</blockquote>
<blockquote>
<p><strong>Systems are seductive</strong> and engineers in particular seem to have a predisposition for them. They promise to do a job faster, better, and more easily than you could do it by yourself or with a less specialized system.</p>
</blockquote>
<blockquote>
<p><strong>Almost anything is easier to get into than out of</strong>. When we introduce new systems, new tools, new lines of code, we’re with them for the long haul. It’s like a baby that doesn’t grow up.</p>
</blockquote>
<p>Sharp observations (and a nudge for me to learn more about <a href="https://en.wikipedia.org/wiki/Systemantics">Gall’s work</a>).</p>
<p>In a nutshell, new systems mean new problems. <a href="/how-complex-web-systems-fail-part-2">Change introduces new forms of failure</a>. We should therefore think twice before writing code or adding third-party services (which are systems too and require developing integration code).</p>
<h2 id="questions-questions-questions">Questions, questions, questions</h2>
<p>This brings me back to Nomad and the general question of whether using a hosted service – assuming there is one – would indeed be better than operating such a system ourselves?</p>
<p>The answer is, of course, that <em>it depends</em> on the circumstances. It’s not always better to use SaaS products just as it’s not always better to operate systems in-house. There are no absolutes – you must weigh the tradeoffs.</p>
<p>At this point, you should ask yourself a lot of questions:</p>
<ul>
<li>
<p>Does it make sense to pay another company for providing service X? After all, it’s <em>their</em> core business, not yours. It’s safe to assume they’re much more skilled in technology X.</p>
</li>
<li>
<p>Alternatively, would it be a good idea to operate the system in-house and take the burden of automation, monitoring, backups, bug fixing, updates, etc.? What other potential gains do you lose when choosing this alternative (opportunity cost)?</p>
</li>
<li>
<p>Do you avoid using already existing software and tend towards reinventing the wheel for the wrong reasons? Beware of the <a href="https://en.wikipedia.org/wiki/Not_invented_here">NIH syndrome</a>!</p>
</li>
<li>
<p>Are you afraid to give up control and become dependent on a vendor? Do you trust the other company to do the right thing? Trust is paramount.</p>
</li>
<li>
<p>What services (if any) are there on the market? Is there a good fit in terms of features, licensing, security, SLA, support, etc.? Spend some time evaluating the available options. Do a quick spike if a product sounds promising. <a href="/chaos-engineering-101">Run chaos experiments</a> to verify your assumptions.</p>
</li>
<li>
<p>What are the total costs of using an existing service compared to hosting it all yourself?</p>
</li>
<li>
<p>Could it be that you <a href="http://danluu.com/sounds-easy/">underestimate the time</a> it takes to build and run a similar system? Features are often much more complicated than we realize.</p>
</li>
<li>
<p>Will it be difficult to integrate with the external service? (Applying reliability design patterns such as timeouts and exponential backoff is an article of its own.) Conversely, what about getting rid of the dependency again? (The software is only done when it’s <em>deleted</em>.)</p>
</li>
</ul>
<p>As you can tell, I’m biased. I prefer using a hosted service provider because that’s how we do things at Jimdo. Operating a system is always our plan B, never plan A. This strategy has been working very well so far.</p>
<h2 id="solving-the-right-problem">Solving the right problem</h2>
<p>Let’s wrap this up with a little reminder. Before deciding whether to run a system or not, I want you to step back for a moment and think again <a href="http://wiki.c2.com/?YouArentGonnaNeedIt">if you actually need more software</a> to solve a problem or if you can somehow do without it.</p>
<p>In <em>Rework</em>, the business book by Jason Fried and David Heinemeier Hansson that is unlike any other book I know, they write:</p>
<blockquote>
<p>Small is not just a stepping-stone. Small is a great destination in itself. […] expenses, rent, <strong>IT infrastructure</strong>, furniture, etc. These things don’t just happen to you. You decide whether or not to take them on.</p>
</blockquote>
<p>We only started looking into running periodic batch jobs on our platform when other development teams kept asking for it. We’ve had the <em>need</em> for such a system, so we took it on.</p>
<p><strong>At the end of the day, it’s not only about solving the problem right, but it’s also about solving the right problem.</strong></p>
Always leave the campground cleaner than you found it2016-11-16T00:00:00+01:00https://sharpend.io/always-leave-the-campground-cleaner-than-you-found-it<p><img src="/assets/images/campground.jpg" alt="" /></p>
<p>We’re currently joining another infrastructure team at Jimdo (let’s call it a restructuring measure). Among other things, this process involves merging the digital Kanban boards of both teams into one – what a great opportunity to go through our backlog and recklessly close tickets that are duplicates, done, or obsolete for one reason or another.</p>
<p>While studying our backlog, I noticed a couple of things:</p>
<ol>
<li>We have too many tickets in our backlog (120+ before the cleanup).</li>
<li>Ideas are worth nothing unless executed.</li>
<li>Things we thought were important turned out <a href="https://m.signalvnoise.com/constraints-only-work-if-they-hurt/">not to matter at all</a>.</li>
</ol>
<p>More specifically, we never came around to complete some code refactoring tasks, despite the fact that several of these tickets are labeled “easy pick”.</p>
<p>Why is that? Is it a good idea to postpone refactorings? And if not, what’s the alternative?</p>
<p>Before trying to answer these questions, let’s look at the main reason we need to refactor our code in the first place.</p>
<h2 id="broken-windows">Broken windows</h2>
<p>In “The Pragmatic Programmer”, there’s a chapter called <a href="https://pragprog.com/the-pragmatic-programmer/extracts/software-entropy">Software Entropy</a>. In a nutshell, the chapter makes the point that entropy – the amount of disorder in a system, which tends to a maximum – is the reason that <em>all</em> projects run the risk of decaying during their lifetime. Yet, there are teams that “successfully fight nature’s tendency toward disorder” and manage to get software rot – technical debt – under control.</p>
<p>But how do they achieve this?</p>
<p>The key realization here is that “<strong>neglect</strong> accelerates the rot faster than any other factor”. Put another way: living with bad code and poor design decisions is likely to lead to even more bad code and poor design decisions.</p>
<p>This understanding is at the heart of the <a href="https://en.wikipedia.org/wiki/Broken_windows_theory">Broken Windows Theory</a>:</p>
<blockquote>
<p>The broken windows theory is a criminological theory of the norm-setting and signaling effect of urban disorder and vandalism on additional crime and anti-social behavior. The theory states that maintaining and monitoring urban environments to prevent small crimes such as vandalism, public drinking, and toll-jumping helps to create an atmosphere of order and lawfulness, thereby preventing more serious crimes from happening.</p>
</blockquote>
<p>The nice thing about this theory is that it is not only true for crimes such as vandalism, it’s true for software development as well.</p>
<p>Once windows start breaking and nobody cares, more serious crimes will follow. The moment you accept substandard code and inadequate designs, your systems begin to deteriorate (and to slow development down as a result).</p>
<p>It’s a matter of mindset: the more broken windows in your codebase, the more likely people are to think “this code is crap anyway”, and the more “crimes” they’re going to commit. On the other hand, if the code is of high quality, people will probably take extra care not to mess things up.</p>
<p>Therefore, <strong>don’t live with broken windows.</strong></p>
<h2 id="refactor-early-refactor-often">Refactor early, refactor often</h2>
<p>Addressing the issue of broken windows – or in our case, software rot – is the part where I disagree with “The Pragmatic Programmer” to some extent.</p>
<p>I do agree that you should <strong>refactor early and refactor often</strong>. Refactor too late and your productivity will decline until you decide to do something about it, which might be a massive undertaking at that point. So it’s generally a good idea to fix flaws as soon as you discover them – when the cost of change is lowest.</p>
<p>The other side of the coin: refactor too early and you risk making rash design decisions based on, well, guessing. In particular, there’s the danger of <a href="http://wiki.c2.com/?PrematureOptimization">“optimizing before we know that we need to”</a> aka premature optimization.</p>
<p>I take issue with the following paragraph from the book though:</p>
<blockquote>
<p>If there is insufficient time to fix [a broken window] properly, then board it up. Perhaps you can comment out the offending code, or display a “Not Implemented” message, or substitute dummy data instead. Take some action to prevent further damage and to show that you’re on top of the situation.</p>
</blockquote>
<p>Recommending to comment out code is questionable advice at best. Code that has no purpose <a href="/the-power-of-less-code">should be killed</a> to not become a source of distraction, confusion, and communication overhead – another broken window, if you will.</p>
<p>The book also suggests putting refactoring tasks on the schedule if you can’t do them immediately. However, if at all, I’ve only seen this work in practice when done as soon as possible, say, within a week. Our backlog is one, albeit small, example of this going wrong.</p>
<h2 id="incremental-refactoring">Incremental refactoring</h2>
<p>I recently read an <a href="http://ronjeffries.com/xprog/articles/refactoring-not-on-the-backlog/">excellent blog post</a> by Ron Jeffries. In it, he adds to the uncomfortable feeling I had about refactoring tickets. He argues that putting them on the backlog is indeed a <em>bad idea</em>, especially when you have a lot of refactoring to do to get back to a “clean field”.</p>
<p>Ron writes:</p>
<blockquote>
<p>We took many weeks to get the code this bad, and we’ll surely not get that many weeks to fix it. […] A big refactoring session is hard to sell, and if sold, it returns less than we hoped, after a long delay.</p>
</blockquote>
<p>What he suggests instead is to <strong>improve the code where we work</strong>:</p>
<blockquote>
<p>We take the next feature that we are asked to build, and instead of detouring around all the weeds and bushes, we take the time to clear a path through some of them. Maybe we detour around others. We improve the code where we work, and ignore the code where we don’t have to work. We get a nice clean path for some of our work. Odds are, we’ll visit this place again: that’s how software development works.</p>
</blockquote>
<p>This type of incremental refactoring reminds me of yet another programming wisdom…</p>
<h2 id="the-boy-scout-rule">The Boy Scout Rule</h2>
<p><a href="https://www.oreilly.com/library/view/97-things-every/9780596809515/ch08.html">The Boy Scout Rule</a> says:</p>
<blockquote>
<p>Always leave the campground cleaner than you found it.</p>
</blockquote>
<p>No matter who’s responsible for the mess, try to improve the environment for the next group.</p>
<p>I hope that now, at the end of this article, you can see how following that simple rule can make a huge difference to the evolution of our production systems – and to the teams building and maintaining them <em>together</em>.</p>
<p><strong>“Boy Scouting” and an aversion to broken windows have been crucial in keeping the technical debt of our PaaS under control – even more so now that I know that refactoring tickets have no place in our backlog.</strong></p>
From Zero to Staging and Back2016-11-02T00:00:00+01:00https://sharpend.io/from-zero-to-staging-and-back<p>When I joined Jimdo’s Werkzeugschmiede team in August 2015, the first major task I took on was building a <em>staging environment</em> for <a href="https://speakerdeck.com/mlafeldt/a-journey-through-wonderland">Wonderland</a>, our in-house PaaS for microservices. We’re an internal service provider offering other teams a platform for running production services. As such, we take a great interest in the uptime of our infrastructure.</p>
<p>Since the very beginning of Wonderland, we knew that an isolated test environment matching production as closely as possible would give us more confidence to experiment, fix bugs, and implement new features. And indeed, having a pre-production environment for testing before deploying to production turned out to be invaluable – a safety net making the whole deployment process a lot less scary.</p>
<p>What follows is a detailed account of Wonderland’s staging environment: what it looks like, how we built it, and what we’ve learned since then.</p>
<h2 id="pair-programming">Pair programming</h2>
<p>Early on we decided to create the much-needed staging environment by pair programming. For me, working on a setup that is supposed to mirror production, and doing this with a coworker who knows Wonderland inside out, was an excellent way to learn about the platform and its different components.</p>
<p>I was able to ask questions when something was unclear and, at the same time, contribute my own ideas whenever I felt like it. This way, we created <a href="/fast-feedback-is-everything">a fast feedback loop</a> that not only helped me find my way through Wonderland, but also learn more about its creators – my new colleagues – and their modus operandi.</p>
<p>Taken all together, I highly recommend pairing for onboarding new team members, even if you don’t have the luxury of building a production-like environment from scratch.</p>
<h2 id="one-account-per-environment">One account per environment</h2>
<p>Wonderland’s infrastructure runs on AWS. Rather than using a single VPC for both production and staging, we agreed to operate a dedicated AWS account per environment. This setup effectively isolates environments from one another. Most importantly, it prevents changes done in staging – whether intentionally or by mistake – from affecting production.</p>
<p>Other advantages of having one AWS account per environment include:</p>
<ul>
<li>An easier understanding of the (cloned) infrastructure</li>
<li>Simpler automation code with fewer environment-specific exceptions</li>
<li>Finer access control on VPC level (no more fiddling around with subnets)</li>
<li>No naming collisions of non-VPC resources</li>
<li>Effortless tracking of costs per environment</li>
<li>Ability to opt-in to AWS features outside of prod first</li>
</ul>
<p>On the downside, working with multiple AWS accounts makes credential management a bit more involved. To make up for this, we’ve been using <a href="https://github.com/Luzifer/awsenv">awsenv</a> and LastPass to quickly switch between accounts. (Of course, it’s all for naught <a href="/writing-your-first-postmortem">if one forgets to use those tools</a>…)</p>
<p>We actually took this separation one step further and also created additional “stage” accounts for <em>all</em> hosted services we rely on every day, such as Papertrail and Quay. The overhead has been worth it.</p>
<h2 id="automate-all-the-things">Automate all the things</h2>
<p>We spent a lot of time automating the setup of our staging environment. We managed to get to the point where we could run <code class="language-plaintext highlighter-rouge">make stage</code> in our <code class="language-plaintext highlighter-rouge">github.com/Jimdo/wonderland</code> repository and Ansible would take care of everything, from bootstrapping our ECS cluster to provisioning Jenkins – our central state enforcer – to starting essential microservices of our PaaS.</p>
<p>To achieve that, we took the existing Ansible playbooks and CloudFormation templates for production and adapted them for use in staging. This meant we had to:</p>
<ul>
<li>Replace hardcoded parameters like URLs and secrets</li>
<li>Implement missing automation steps (some prod resources had been clicked)</li>
<li>Address any issues that came up along the way (two words: eventual consistency)</li>
</ul>
<p>It was also at this point that we decided to leverage standard make targets like “stage” or “prod” across projects. The following paragraph from <a href="https://medium.com/@jlouis666/how-to-build-stable-systems-6fe9dcf32fc4">How to build stable systems</a> sums it up very well:</p>
<blockquote>
<p>All projects, language notwithstanding, use the same tool for configuring and building themselves: make(1). Make can call into the given languages choice of build tool, but the common language for continuous integration and deployment is make(1). Use the same make targets for all projects in the organization. This makes it easy to onboard new people and they can just replay the work the CI tool is doing. It also “documents” how to build the software.</p>
</blockquote>
<p><strong>Even after automating all the things, there’s only one way to find out if our code works – and continues to work – as expected: creating staging from scratch, again and again.</strong></p>
<h2 id="destroy-all-the-things">Destroy all the things</h2>
<p>In addition to <code class="language-plaintext highlighter-rouge">make stage</code>, we also implemented the inverse operation, <code class="language-plaintext highlighter-rouge">make destroy-stage</code>, to deprovision staging completely. This process boils down to deleting all CloudFormation stacks and other resources created by Ansible – in reverse order of creation.</p>
<p>Tearing down CloudFormation stacks is usually straightforward. However, we sometimes have to shell out to the AWS CLI because CloudFormation is very slow when it comes to adding new resources. This can lead to dependencies that are hard to remove. And even when Ansible does provide a particular AWS module, there’s no guarantee that the “absent” state is implemented correctly, making the dreaded CLI our only option.</p>
<p>Once <code class="language-plaintext highlighter-rouge">make destroy-stage</code> did the trick, we were able to bootstrap staging from scratch, which in turn allowed us to verify that our infrastructure code does the right thing when starting from a blank slate.</p>
<p>To further automate things, we created a Jenkins job in prod to destroy staging every Friday night and another one to rebuild it on Monday morning.</p>
<h2 id="drawbacks-and-improvements">Drawbacks and improvements</h2>
<p>While I’m happy with what we’ve achieved so far, there’s still room for improvement. Here are some of the challenges we’ve seen:</p>
<ul>
<li>
<p>Having a single staging environment for all four members of our team means that we need to coordinate testing in a few cases. Nobody likes to wait, especially not me. One solution would be to split up those Jenkins jobs and/or the backend systems they target.</p>
</li>
<li>
<p>It currently takes at least 5 hours to rebuild staging. We’ve already outsourced the building and storage of Docker images to Travis and Quay. We could further accelerate the process, for example by baking AMIs of our cluster instances. Again, nobody likes to wait.</p>
</li>
<li>
<p>Unsurprisingly, we experienced a couple of problems caused by broken/missing external dependencies. One that comes to mind is <a href="https://github.com/docker/docker/issues/23203">Docker’s APT repository</a>. Yes, mirrors would certainly help. We started using GitHub releases for hosting artifacts whenever possible.</p>
</li>
<li>
<p><a href="https://queue.acm.org/detail.cfm?id=2353017">Allspaw is right when he says</a> that “[testing outside production] is incomplete because some behaviors can be seen only in production, no matter how identical a staging environment can be made”. We’ve learned this the hard way. Staging is a safety measure – no more, no less.</p>
</li>
<li>
<p>To be honest, the automated weekly rebuild of staging has caused us a lot of trouble and extra work lately. Sometimes the Jenkins job fails after running into API limits. Other times the job orchestration goes wrong due to network issues (or because Jenkins happens to have a bad day?). In any case, we need to make the process more reliable again.</p>
</li>
</ul>
<p>For more on this topic, I recommend reading my other post: <a href="/if-it-hurts-do-it-more-often">If it hurts, do it more often</a>.</p>
The Myth of the Root Cause: How Complex Web Systems Fail (Scalyr)2016-10-20T00:00:00+02:00https://sharpend.io/the-myth-of-the-root-causeThe Power of Less Code2016-10-19T00:00:00+02:00https://sharpend.io/the-power-of-less-code<p><img src="/assets/images/garbage-truck.jpg" alt="" /></p>
<p>These days, I spend a lot of time automating, debugging, and fixing the various components that make up Jimdo’s internal PaaS, which ultimately serves the 15+ million websites of our customers.</p>
<p>My team knows that <em>reliable</em> processes are essential. To that end, we continuously make our infrastructure code more robust, tune our monitoring and alerting setup, and <a href="/chaos-engineering-101">run GameDay exercises</a> on a regular basis.</p>
<p>We’re the team that has been building and running the PaaS – <em>we own it</em> – and this is unlikely to change soon. For that reason, we strive to keep the platform as simple as possible, while being aware that a certain amount of complexity is necessary for our systems to do anything useful.</p>
<p>Of course, we still do enjoy creating new things from time to time; we’re programmers, after all. For example, we love to build new automation tools and add service features to improve observability and scalability.</p>
<p>On the other hand – and this is the crucial point – we also care a lot about doing the opposite:</p>
<ul>
<li>Deleting superfluous code</li>
<li>Removing features</li>
<li>Deleting documentation</li>
<li>Closing stale pull requests</li>
<li>Throwing away Git branches</li>
<li>Destroying cloud resources</li>
<li>Decommissioning projects</li>
</ul>
<p><strong>In other words, we try hard to get rid of anything that isn’t needed.</strong></p>
<p>I know that more often than not, reality looks somewhat different:</p>
<ul>
<li>Projects are on a tight schedule and cleaning up is not considered a priority.</li>
<li>The “DevOps department” has lost sight of server costs and no one has an eye on unused cloud resources.</li>
<li>Developers, in general, are more keen to work on new features, whereas deleting unfinished experiments isn’t half as much fun.</li>
</ul>
<p>Sounds familiar, doesn’t it?</p>
<p>While the benefits of terminating unused cloud servers should be obvious to anyone who’s paying the bills, it might not be so clear otherwise. What exactly is so bad about developing more features, more tools, and therefore more code? Why bother deleting anything?</p>
<p><strong>The fact of the matter is that code is a liability. You and your teammates are responsible for each and every line of code you produce.</strong></p>
<p>As Jeff Atwood put it so eloquently in his popular blog post, <a href="http://blog.codinghorror.com/the-best-code-is-no-code-at-all/">The Best Code is No Code At All</a>:</p>
<blockquote>
<p>Every new line of code you willingly bring into the world is code that has to be debugged, code that has to be read and understood, code that has to be supported. Every time you write new code, you should do so reluctantly, under duress, because you completely exhausted all your other options.</p>
</blockquote>
<p>On top of that, the <a href="http://shop.oreilly.com/product/0636920030355.do">Lean Enterprise</a> book offers these valuable insights:</p>
<blockquote>
<p>At first, we should propose solutions that don’t involve writing code […] Software development should always be a last resort, because of the cost and complexity of building and maintaining software.</p>
</blockquote>
<blockquote>
<p>Only spend time and effort on test automation for products or features once they have been validated. Test automation for experiments is wasteful.</p>
</blockquote>
<blockquote>
<p>Our most productive people are those that find ingenious ways to avoid writing any code at all.</p>
</blockquote>
<p>Less code means less complexity, which means less bugs, which means less unexpected outcomes in production. Remember: <a href="/simplicity-a-prerequisite-for-reliability">simplicity is a prerequisite for reliability</a>. The more complex a system, the more difficult it is to build a mental model of the system, and the harder it becomes to operate and debug it.</p>
<p>Code that has no purpose is a major source of distraction, confusion, and communication overhead (<em>“Hey Mathias, what’s the point of this function parameter we don’t use anywhere?”</em>). It’s poor practice to comment out unused code, or worse, to gate it with a feature flag. Today’s version control systems make it easy to revert any changes; there’s no reason not to remove dead code and other bloat, such as outdated documentation and pull requests that haven’t been updated in months. In the wise words of Kent Beck: <a href="https://twitter.com/KentBeck/status/525719404643094528">complete it or delete it</a>.</p>
<p>(Just to be clear: it’s totally possible to produce fewer lines of code by writing clever code. <a href="http://www.codethinked.com/dont-be-clever">Don’t be clever either</a>. It will make the code even harder to understand.)</p>
<p>Deleting many – sometimes hundreds or thousands – lines of code is indeed extremely satisfying. It’s a worthwhile investment of time, one that comes very close to the joy of building, at least for me.</p>
<p>A couple of weeks ago, we finished the migration from Dkron to <a href="https://www.nomadproject.io/">Nomad</a> for running all periodic batch jobs of our PaaS. It was a lengthy migration to say the least (operating Nomad in a highly-available fashion in AWS isn’t trivial). So I did one particular thing to keep me motivated: I created a pull request in which I prepared all changes to our infrastructure code required to decommission Dkron and its dependencies – weeks before <a href="https://twitter.com/mlafeldt/status/778971852366635008">actually pulling the plug</a> on the stack and throwing away the 1000 lines of code for provisioning it.</p>
<p>Believe it or not, but this pull request – the prospect of reducing the overall complexity of our infrastructure – kept me excited about the project until the very end.</p>
<p>And yet there’s a much better solution we would have preferred: <em>not</em> doing all this work in the first place, but rather outsource the task of running periodic jobs to a hosted service provider with a paid support plan. Unfortunately, there’s no Nomad Enterprise (or something comparable that fits our needs) yet, so for now we’re left with the operational burden.</p>
<p>I hope this will change because <strong>the best code is no code at all</strong>.</p>
<p>Let’s acknowledge this fact by saying “no” to code more often than “yes”.</p>
On Finding Root Causes2016-10-05T00:00:00+02:00https://sharpend.io/on-finding-root-causes<p><img src="/assets/images/hay.jpg" alt="" /></p>
<p><a href="/writing-your-first-postmortem">In my previous article</a> I introduced you to postmortems – what they are, why you should conduct them, and how to get started writing your own postmortem documents.</p>
<p>As a quick recap, a postmortem is a written record of an incident documenting its impact, what caused it, the actions taken to mitigate or fix it, and how to prevent it from happening again. In a broader sense, postmortems are a great tool for a company or organization to learn from failure.</p>
<p>One of the big questions a postmortem has to address is: What has caused the incident? – What’s the reason the system failed the way it did?</p>
<p>At first glance, finding the <em>root cause</em> – the initiating cause that led to an outage or degradation in performance – seems to be the rational thing to do. For system owners, knowing <em>who</em> or <em>what</em> is responsible for an incident appears to be a desirable goal. Otherwise, how else should they implement appropriate countermeasures?</p>
<p><strong>In reality, however, trying to attribute an incident to a root cause in hindsight is not only impossible – it is fundamentally wrong.</strong></p>
<p>This and most of the lessons that follow have their origin in Richard Cook’s paper “How Complex Systems Fail”. I already devoted <a href="/how-complex-web-systems-fail-part-1">a two-part series</a> to his seminal work, but there’s so much more to learn from it, especially when it comes to postmortems. I think you’ll agree.</p>
<h2 id="there-is-no-single-root-cause">There is no single root cause</h2>
<p>In complex systems, such as web systems, there is no root cause. Single point failures alone are not enough to trigger an incident. Instead, incidents require <em>multiple contributors</em>, <a href="http://www.kitchensoap.com/2012/02/10/each-necessary-but-only-jointly-sufficient/">each necessary but only jointly sufficient</a>. It is the combination of these causes – often small and innocuous failures – that is the prerequisite for an incident.</p>
<p><strong>As a consequence, we can’t isolate a single root cause.</strong></p>
<p>One reason we tend to look for a single, simple cause of an outcome is because the failure is too complex to keep it in our head. Thus we oversimplify without really understanding the failure’s nature and then blame particular, local forces or events for outcomes.</p>
<p>One of the things I like about the <a href="https://gist.github.com/mlafeldt/6e02ea0caeebef1205b47f31c2647966">postmortem template</a> I mentioned last time is that it says “Root Cause<em>s</em>”, not “Root Cause”. For me, that’s a testament to the fact that you need to look deeper if you only have a single root cause.</p>
<p>But even “Root Causes” might not be the best term, as Andy Fleener has pointed out to me <a href="https://twitter.com/andyfleener/status/778586201313845250">on Twitter</a>:</p>
<blockquote>
<p>I definitely prefer “Contributing Conditions” over Root Causes though. Cause can only be constructed with the benefit of hindsight</p>
</blockquote>
<p>That’s a good point that made me consider modifying our own postmortem template as well.</p>
<h2 id="hidden-biases-in-our-thinking">Hidden biases in our thinking</h2>
<p><em>Hindsight bias</em> continues to be the main obstacle to incident investigation. This cognitive bias, also known as the knew-it-all-along effect, describes the tendency of people to overestimate their ability to have predicted an event, despite the lack of objective evidence.</p>
<p><strong>Indeed, hindsight bias makes it impossible to accurately assess human performance after an incident.</strong></p>
<p>A similar but different cognitive error is <em>outcome bias</em>, which refers to the tendency to judge a decision by its eventual outcome. It’s important to understand that <em>every</em> outcome – successful or not – is the result of a gamble. The overall complexity of our web systems always poses unknowns. We can’t eliminate uncertainty.</p>
<p>After an incident has occurred, a postmortem might find that the system has a history of “almost incidents” and that operators should have recognized the degradation in system performance before it was too late. That’s an oversimplified view though. System operations are <em>dynamic</em>. Failing components and human beings are being replaced all the time. Attribution is not that simple.</p>
<p>We therefore need to be cautious of <a href="https://betterhumans.coach.me/cognitive-bias-cheat-sheet-55a472476b18">hindsight bias and its friends</a>, and never ignore other driving forces, especially production pressure, when looking for root causes after an incident has occurred.</p>
<h2 id="human-error-is-never-a-root-cause">Human error is never a root cause</h2>
<p>It’s the easiest thing in the world to point the finger at others when things go wrong. And unfortunately, many companies still blame people for mistakes when they should really blame – and fix – their broken processes.</p>
<p>Blameless postmortems only work if we assume that everyone involved in an incident had good intentions. This ties in with the <a href="http://retrospectivewiki.org/index.php?title=The_Prime_Directive">Retrospective Prime Directive</a> (a postmortem is a special form of a retrospective), which says:</p>
<blockquote>
<p>Regardless of what we discover, we understand and truly believe that everyone did the best job they could, given what they knew at the time, their skills and abilities, the resources available, and the situation at hand.</p>
</blockquote>
<p><strong>Human error is NOT a root cause.</strong></p>
<p>We should rather look for flaws in systems and processes – the causes contributing to failure – and implement measures so that the same issues don’t happen again. It requires <em>systems thinking</em>, which focuses on cyclical rather than linear cause and effect, to view the system as a whole in order to find out how it drifted into failure – both at a technical and organizational level.</p>
<p>Here’s one of my favorite passages on the topic, taken from <a href="https://landing.google.com/sre/book.html">the SRE book</a>:</p>
<blockquote>
<p>When postmortems shift from allocating blame to investigating the systematic reasons why an individual or team had incomplete or incorrect information, effective prevention plans can be put in place. You can’t “fix” people, but you can fix systems and processes to better support people making the right choices when designing and maintaining complex systems.</p>
</blockquote>
<p><a href="https://www.unwiredcouch.com/2014/08/04/human-error-getting-off-the-hook.html">Does this mean that operators are off the hook? No, not at all.</a> They’re the ones with the most knowledge surrounding the incident. For example, they know first-hand how the system failed in surprising ways. Hence, they’re responsible for finding ways to make the system more resilient – including writing a postmortem.</p>
<h2 id="practical-example">Practical example</h2>
<p>Let’s wrap this up with an example from an actual postmortem.</p>
<p>Last time I told you a story about a recent outage at Jimdo. In a nutshell: To fix a broken deployment of our API service, I wanted to delete the corresponding ECS service in our AWS staging account. Unfortunately, I actually removed the service in our <em>production</em> account, causing our API to be down for half an hour. <a href="https://twitter.com/mlafeldt/status/772769070555037696">Oops!</a></p>
<p><a href="https://gist.github.com/mlafeldt/01953b48b0be5fea34c11a8a47d1e7f4">In the postmortem that followed</a>, we identified two root causes:</p>
<ol>
<li>The tool we’re using to log into AWS accounts made it difficult to figure out in which account one operates. This contributed to deleting the ECS service in the wrong account. (<a href="https://github.com/Luzifer/awsenv/issues/11">We subsequently improved the tool</a> to show more helpful environment information.)</li>
<li>The component that deploys services to our PaaS didn’t notice that the underlying ECS service was gone and still tried to update the (non-existing) service – which failed. This made it impossible to re-deploy the API service without manually removing all remaining pieces of the original service. (We’ve since fixed the operation to be idempotent.)</li>
</ol>
<p>I’m sure that if we had looked closer, we would have found more root causes contributing to the failure, but we stopped here, made our homework, and moved on.</p>
<p>One more time: <strong>You can’t fix people, but you can fix systems and processes to better support them.</strong></p>
<p>Keep this in mind when you write your next postmortem.</p>
Writing Your First Postmortem2016-09-21T00:00:00+02:00https://sharpend.io/writing-your-first-postmortem<p>I’m one of the operators of <a href="https://speakerdeck.com/mlafeldt/a-journey-through-wonderland">Wonderland</a>, Jimdo’s in-house PaaS for microservices.</p>
<p>Two weeks ago, on September 5, I did something embarrassing at work.</p>
<p>We were debugging a broken deployment of our central API service. This API is nothing less than the entry point for managing all container-based services running on our platform, including most of our own system services (by virtue of dogfooding).</p>
<p>In an attempt to fix the problem we were experiencing – our API service failed to scale to a certain number of replicas – I deleted what I believed to be a duplicate instance of the corresponding ECS service in the AWS Management Console…</p>
<p>That turned out to be a mistake.</p>
<p>Instead of performing this action in our AWS staging account, as I intended to do, I accidentally deleted the ECS service in our <em>production</em> account. Worse, I did not delete some duplicate; it was the real thing.</p>
<p>To make a long story short, this blunder caused our API to be down for 31 minutes, mainly because it took us very long to figure out how to redeploy the broken API service.</p>
<p>Guess what I did immediately after resolving the incident and telling our users the good news?</p>
<p><strong><a href="https://gist.github.com/mlafeldt/01953b48b0be5fea34c11a8a47d1e7f4">I started writing a postmortem</a>. Not because I had to, but because I know that postmortems are the ultimate tool to learn from incidents.</strong></p>
<h2 id="postmortems-101">Postmortems 101</h2>
<p>So, what is a postmortem?</p>
<p>A postmortem is a written record of an incident. Among other things, it documents the incident’s impact, what caused it, the actions taken to mitigate or fix it, and how to prevent it from happening again.</p>
<p>I will tell you more about the ingredients of a good postmortem later in this article. For now, I want you to understand that fixing the underlying issue(s) of an incident is important – but not enough. We also need a formalized process to learn from these incidents.</p>
<p>That’s what postmortems are for. Postmortems help us understand why incidents happen and how we might better prepare our systems for the future. We share this knowledge with other teams or, better yet, <a href="https://github.com/danluu/post-mortems">make postmortems public</a> so that more people can benefit from them – and see that we actually <em>care</em>.</p>
<p><strong>Failure isn’t a disaster, it’s a learning opportunity.</strong></p>
<p>Similar to Chaos Engineering, conducting postmortems requires a fundamental <a href="/chaos-engineering-a-shift-in-mindset">shift in the mindset</a> of managers and employees, if not whole companies. And like Chaos Engineering, postmortems have the potential to make a company more resilient as a whole.</p>
<p>We need to embrace failure for postmortems to work. This includes having <a href="https://codeascraft.com/2012/05/22/blameless-postmortems/">blameless postmortems</a> and no finger-pointing when looking for root causes. (By the way, human error is <em>never</em> a root cause, but more on that in the second part of this article.)</p>
<p>When to write a postmortem? That’s up to you. A good rule of thumb is to write one if an incident has an immediate impact on users (downtime or data loss) or if a stakeholder asks for it. Putting together a decent postmortem often takes a couple of hours, but the effort is usually worth it! Just remember to start the work as soon as possible, with events still fresh in mind.</p>
<h2 id="postmortem-template">Postmortem template</h2>
<p>Inspired by another infrastructure team at Jimdo, we started using the <strong><a href="https://gist.github.com/mlafeldt/6e02ea0caeebef1205b47f31c2647966">Example Postmortem from the SRE book</a></strong> as a template for the postmortems we do for Wonderland.</p>
<p>If you follow the link, you’ll find a Markdown version of said template. I created it from the PDF book for two reasons: First, I wanted to share our postmortems as part of our standard Wonderland documentation, so that our users (other Jimdo teams) can easily find them. Second, all of our development is based on GitHub and we’re used to writing and reviewing Markdown files. In other words, my goal was to reduce barriers to reading, writing, and publishing our postmortems.</p>
<p>As far as I can tell, this initiative was a success. I’ve been astonished how quickly my teammates have adopted the new template. Ever since I published the first postmortem in this manner, they’ve been eager to do the same after a new incident has occurred. I’m aware that this is, for the most part, a matter of having the right mindset. However, it certainly doesn’t hurt to make the process more pleasant for everyone involved.</p>
<p>Publishing more postmortems ultimately means being more transparent about failures, which in turn builds trust in our team and makes our platform more reliable. We’re an internal service provider after all.</p>
<h2 id="your-first-postmortem">Your first postmortem</h2>
<p>Now I encourage you to <a href="https://gist.github.com/mlafeldt/6e02ea0caeebef1205b47f31c2647966">use the template</a> as a foundation for your next – or perhaps first – postmortem. Give it a read. If there wasn’t an incident in last few days (I hope so!), think of the last time you had to deal with an outage. Then go through the different sections and try to fill in the blanks. Be consistent. Use the active voice throughout the document. Settle on a time zone and format. As with most templates, feel free to customize it to your own needs.</p>
<p>Here’s a summary of what each section in the template postmortem is about, including short examples:</p>
<ul>
<li><strong>Title</strong> – Name of the postmortem, e.g. “Deployer API Outage Postmortem”</li>
<li><strong>Date</strong> – When the incident happened, e.g. “2016-09-05”</li>
<li><strong>Authors</strong> – List of people who wrote the postmortem (GitHub handles work fine)</li>
<li><strong>Status</strong> – Current status of the postmortem, e.g. “Complete, action items in progress”</li>
<li><strong>Summary</strong> – A one-sentence summary of the incident, usually something like “Service X was down for N minutes due to Y”</li>
<li><strong>Impact</strong> – The incident’s impact on customers and, if known, revenue or reputation, e.g. “Users were unable to do X while service Y was unavailable from 09:29 to 10:00 UTC”</li>
<li><strong>Root Causes</strong> – A list of causes that have contributed to the incident (I’ll cover root causes in part 2 of this article)</li>
<li><strong>Trigger</strong> – What triggered the outage? e.g. “Merging pull request X which started the rollout of broken software Y”</li>
<li><strong>Resolution</strong> – The action(s) that mitigated and resolved the outage, e.g. “Disabling feature X helped to mitigate the problem. Rolling back to version Y resolved it.”</li>
<li><strong>Detection</strong> – How the problem was noticed, e.g. “Pingdom detected that service X was down and paged on-call via PagerDuty”</li>
<li><strong>Actions Items</strong> – A list of actions taken (with links to GitHub issues) to mitigate or resolve the incident, and to prevent it from recurring</li>
<li><strong>Lessons Learned</strong> – What went well? What went wrong? And what was sheer luck?</li>
<li><strong>Timeline</strong> – A detailed timeline of the events related to the incident</li>
<li><strong>Supporting Information</strong> – Additional graphs, screenshots, command output, etc.</li>
</ul>
<p>Postmortems are a collaborative effort thriving on feedback. Make sure to share first drafts internally with your team. Once the review is complete, share the postmortem with as many people as possible.</p>
<p>Always keep in mind: <strong>Failure isn’t a disaster, it’s a learning opportunity for the whole company.</strong></p>
<hr />
<p>That’s the end of part 1. <a href="/on-finding-root-causes">In part 2</a>, I’m going to dive into the wonderful world of <em>root causes</em>, probably the most important – and most difficult – element in conducting a postmortem.</p>
Systems blindness and how we deal with it2016-09-07T00:00:00+02:00https://sharpend.io/systems-blindness-and-how-we-deal-with-it<p><img src="/assets/images/systems-blindness.jpg" alt="" /></p>
<p>In what must have been an impulse purchase, I bought a paperback copy of Daniel Goleman’s book “Focus: The Hidden Driver of Excellence” a couple of weeks ago. I’m still only halfway through this hard-to-follow mishmash of ideas, all supposedly related to the overall theme of focus in a distracted world. There’s this one chapter in the book, however, that stands out to me. The chapter is titled “System Blindness” and it provides enough insights – food for thought – that I don’t regret reading the book. Here’s what I learned.</p>
<h2 id="mental-models-and-outages">Mental models and outages</h2>
<p>Systems are invisible to our eyes. We try to understand them indirectly through mental models and then perform actions based on these models.</p>
<p>As I wrote before, <a href="/simplicity-a-prerequisite-for-reliability">simplicity is a prerequisite for reliability</a>. The more complex a system, the more difficult it is to build a mental model of the system, and the harder it becomes to operate and debug it.</p>
<p>As a matter of fact, the majority of outages are self-inflicted. We’re thinking about a change we’re going to make, but we don’t necessarily anticipate the negative consequences it might have on the system as a whole. We push a bad configuration or deploy a buggy Docker image and all of a sudden the website goes down. It has happened to all of us. I, for one, have certainly caused <a href="https://twitter.com/mlafeldt/status/772769070555037696">my fair share of outages</a>.</p>
<p>For lack of a better word, I always used to refer to <a href="/unintended-consequences">“unintended consequences”</a> whenever I talked about these unexpected drawbacks that go along with complexity.</p>
<p>Then came Goleman’s book, which introduced me to the term <strong>systems blindness</strong>.</p>
<h2 id="what-is-systems-blindness">What is systems blindness?</h2>
<blockquote>
<p>Systems blindness is the main thing we struggle in our work. What we think of as “side effects” are misnamed. […] In a system there are no side effects – just effects, anticipated or not. What we see as “side effects” simply reflect our flawed understanding of the system. In a complex system […] cause and effect may be more distant in time and space than we realize.</p>
</blockquote>
<p>According to Goleman, one of the worst results of systems blindness occurs when we implement a strategy to fix a problem but ignore the involved system dynamics. Many problems are, unfortunately, too macro or micro for us to notice directly.</p>
<p>One example he gives is that of building more and wider roads to avoid traffic jams, which will eventually lead to even more traffic as people will take advantage of the better travel connections and move further away from urban areas. Another example is global warming. Energy and climate are a system. <em>Everything</em> that we’re doing is part of the healing of that system. It’s a systemic problem; climate meetings and agreements can only do so much.</p>
<p>Likewise, everything that we’re doing affects the success and failure of our software systems. When something doesn’t work as expected, I want you to remember this lesson: <strong>There are no side effects, just effects that result from our flawed understanding of the system.</strong></p>
<h2 id="illusion-of-explanatory-depth">Illusion of explanatory depth</h2>
<p>Another phenomenon related to systems blindness is the “illusion of explanatory depth”.</p>
<p>We often believe we understand how something works when in reality our understanding is superficial at best. In our industry, this illusion becomes apparent when trying to explain <em>in depth</em> how technology X works, where X might be: Kubernetes, the TCP/IP stack, Linux syscalls, AES encryption, consensus over Raft, the list goes on and on.</p>
<p>And even if someone has comprehensive knowledge of certain technologies, there’s still the challenge of grasping the dynamics – the feedback loops – of the larger system in which they’re embedded.</p>
<p>Distributed systems are hard for a reason.</p>
<h2 id="patterns-and-rules">Patterns and rules</h2>
<p>To some degree, biology is to blame for our imperfect systems understanding:</p>
<blockquote>
<p>[In contrast to self-awareness and empathy], there seems to be no dedicated network or circuitry in the brain that gives us a natural inclination toward systems understanding. We learn how to read and navigate systems through the remarkable general learning talents of the neocortex.</p>
</blockquote>
<p>In other words, systems thinking – <a href="http://www.umsl.edu/~sauterv/analysis/bees/">the cure for systems blindness</a> – is a skill that must be learned, just like reading or programming. (Goleman notes that computer games can teach us how to experiment with complex systems.)</p>
<p>At the same time, we humans excel at “detecting and mapping the patterns and order that lie hidden within the chaos of the natural world” because our very survival depends on it.</p>
<blockquote>
<p>We live within extremely complex systems, but engage them lacking the cognitive capacity to understand or manage them completely. Our brain has solved this problem by finding means to sort through what’s complicated via simple decision rules [e.g. trusting other people]</p>
</blockquote>
<p>Our built-in pattern detector is able to simplify complexity into manageable decision rules.</p>
<p>This is a major reason why we, especially in IT, are obsessed with data (Big Data, anyone?). We strive to make the workings of our production systems visible by gathering <em>and</em> curating enough data points, like metrics and logs, that the dynamics of these systems become palpable. Armed with the myriad of supporting tools available today, we look for meaningful patterns within that data – knowing where to focus in a system is key – and take actions based on these patterns.</p>
<p><strong>It is this observability that, in the absence of perfect understanding, helps us deal with the complexity of our systems.</strong></p>
<p>I would like to thank Goleman for making me realize this connection.</p>
The Obvious, the Easy, and the Possible2016-08-24T00:00:00+02:00https://sharpend.io/the-obvious-the-easy-and-the-possible<p><img src="/assets/images/buckets.jpg" alt="" /></p>
<p>Last year, when the show was still running, I used to binge-listen to <em>Work In Progress</em>, the podcast with Jason Fried (founder and CEO of Basecamp) and Nathan Kontny (CEO of Highrise).</p>
<p><a href="https://www.youtube.com/watch?v=9UELVcyTJK4">In one of my favorite episodes</a>, Fried talks about the idea of thinking about product development and interface design in terms of three buckets: the obvious, the easy, and the possible.</p>
<p>In this article, I’m going to explain what this idea is all about and why I think it’s so powerful – even if you’re more of a programmer person and don’t think of yourself as someone who can design web applications.</p>
<h2 id="the-three-buckets">The three buckets</h2>
<p>Whenever you build a product or feature, you have to figure out what matters – and to what degree – and what does not. You then focus development efforts appropriately.</p>
<p>According to Fried, you have to understand which things go in which bucket by asking yourself these three questions:</p>
<ul>
<li>
<p><strong>What needs to be obvious?</strong> The thing(s) people do all the time – the core of the product – should be obvious. On Twitter, for example, it’s obvious how to send a tweet thanks to the big blue <em>Tweet</em> button in the upper right corner. Not everything can be obvious, of course, because screen real estate, attention span, etc. are limited resources. It’s your job to decide.</p>
</li>
<li>
<p><strong>What should be easy?</strong> The things people do frequently, but not always, should be easy. It can be hard to know the difference between the two, though. Byword, <a href="/blog/writing-apps-for-osx">my writing app of choice</a>, makes it easy to export a Markdown document to PDF via the <em>File</em> menu. In a writing app, exporting isn’t something people do all the time, yet it’s done often enough that it should be easy to accomplish.</p>
</li>
<li>
<p><strong>What should be possible?</strong> The things people do sometimes – or rarely – should at least be possible. On GitHub, it’s possible (but not obvious or particularly easy) to create a new OAuth token via <em>Settings -> Personal access tokens -> Generate new token -> Confirm password -> Enter token details</em>. As there tend to be way more things in this bucket than in the other ones, you should ask yourself if making something at all is the right choice.</p>
</li>
</ul>
<h2 id="command-line-interfaces">Command-line interfaces</h2>
<p>You might be wondering why I, a systems guy who’s mostly into web infrastructure, care about design principles. Indeed, the only interfaces I normally “design” are command-line interfaces – the primary means of interaction with the bulk of server software.</p>
<p>But even though designing a web application and creating a CLI are two different beasts, the same principles apply. When developing a command-line tool, you too have to figure out:</p>
<ul>
<li>
<p><strong>What needs to be obvious?</strong> It’s best practice to provide commands for the most common operations. For instance, the Go tool makes it clear that <code class="language-plaintext highlighter-rouge">go build</code> will compile source code whereas <code class="language-plaintext highlighter-rouge">go test</code> will run tests. Most Go programmers use both many times a day, justifying the existence of these commands.</p>
</li>
<li>
<p><strong>What should be easy?</strong> You should offer command-line options (or sub-commands) for actions users do frequently. The <code class="language-plaintext highlighter-rouge">go get</code> command, for example, downloads and installs packages along with their dependencies. When passed the <code class="language-plaintext highlighter-rouge">-d</code> option, it will skip the installation step, providing a convenient way to fetch all of a project’s dependencies.</p>
</li>
<li>
<p><strong>What should be possible?</strong> You may allow users to enable advanced, experimental, or dangerous features via environment variables or lengthy command-line options. Setting the <code class="language-plaintext highlighter-rouge">GO15VENDOREXPERIMENT</code> variable in Go 1.5, for example, will tell the Go tool to resolve dependencies in <code class="language-plaintext highlighter-rouge">vendor/</code> directories. For other use cases, it might be enough to output data in a structured format that’s easy for other tools to process.</p>
</li>
</ul>
<h2 id="conclusion">Conclusion</h2>
<p>No matter if you’re creating a web application, command-line tool, or any other user-facing product, thinking deeply about what needs to be obvious, easy, or possible can mean the difference between building something that merely gets the job done and something that’s a joy to use.</p>
How Complex Web Systems Fail - Part 22016-08-10T00:00:00+02:00https://sharpend.io/how-complex-web-systems-fail-part-2<p><img src="/assets/images/complex-2.jpg" alt="Photo by Frankie McKenzie" /></p>
<p>In his influential paper <a href="http://web.mit.edu/2.75/resources/random/How%20Complex%20Systems%20Fail.pdf">How Complex Systems Fail</a>, Richard Cook shares 18 brilliant observations on the nature of failure in complex systems. <a href="/how-complex-web-systems-fail-part-1">Part 1 of this article</a> was my attempt to translate the first nine of his observations into the context of web systems, i.e., the distributed systems behind modern web applications. In this second and final part, I’m going to complete the picture and cover the other half of Cook’s paper. So let’s get started with observation #10!</p>
<h3 id="10-all-practitioner-actions-are-gambles">10. All practitioner actions are gambles</h3>
<p>Cook notes that all actions we take in response to an accident are just <em>gambles</em>. There are things we believe we know (e.g., because we built the system in a such-and-such way), but conversely, there are also things we don’t know (and even <a href="https://en.wikipedia.org/wiki/There_are_known_knowns">ones we don’t know we don’t know</a>). The overall complexity of our web systems always poses unknowns. We can’t eliminate uncertainty – the guessing of what might be wrong and what might fix it.</p>
<p>As we learned in part 1 of this article, it’s impossible to correctly assess human performance after an accident due to cognitive errors like hindsight bias (see observation #8). A similar but different phenomena is <a href="https://en.wikipedia.org/wiki/Outcome_bias">the outcome bias</a>, well illustrated by Cook:</p>
<blockquote>
<p>That practitioner actions are gambles appears clear after accidents; in general, post hoc analysis regards these gambles as poor ones. But the converse: that successful outcomes are also the result of gambles; is not widely appreciated.</p>
</blockquote>
<h3 id="11-actions-at-the-sharp-end-resolve-all-ambiguity">11. Actions at the sharp end resolve all ambiguity</h3>
<p>More often than not, companies don’t have a clear direction when it comes to “the relationship between production targets, efficient use of resources, economy and costs of operations, and acceptable risks of low and high consequence accidents”, as Cook states. I would even go so far as to say that, in the absence of hard numbers, decisions are made following someone’s gut feeling.</p>
<p>This ambiguity is resolved by actions <em>at the sharp end</em> of the system, successful or not. After a disaster has struck in production, we’ll know, for example:</p>
<ul>
<li>Management’s response to failure</li>
<li>What went well, what went wrong</li>
<li>If we need to hire another Site Reliability Engineer</li>
<li>Whether we should invest in employee training or better equipment</li>
</ul>
<p>In other words, we’re forced to <em>think and decide</em>.</p>
<p>Once again, we need to be cautious of hindsight bias and its friends, and never “ignore the other driving forces, especially production pressure” after an accident has occurred.</p>
<h3 id="12-human-practitioners-are-the-adaptable-element-of-complex-systems">12. Human practitioners are the adaptable element of complex systems</h3>
<p>It’s <em>people</em> that keep web systems up and running by incrementally improving them – adapting them to new circumstances – so that they can survive in production.</p>
<p>The paper lists the following adaptations as examples:</p>
<ul>
<li>Restructuring the system in order to reduce exposure of vulnerable parts to failure.</li>
<li>Concentrating critical resources in areas of expected high demand.</li>
<li>Providing pathways for retreat or recovery from expected and unexpected faults.</li>
<li>Establishing means for early detection of changed system performance in order to allow graceful cutbacks in production or other means of increasing resiliency.</li>
</ul>
<p>It’s surprisingly straightforward to translate this list into best practices in the field of web operations: decoupling of system components, capacity planning, graceful error handling, periodic backups, monitoring, code instrumentation, canary releases, and so on.</p>
<h3 id="13-human-expertise-in-complex-systems-is-constantly-changing">13. Human expertise in complex systems is constantly changing</h3>
<blockquote>
<p>Complex systems require substantial human expertise in their operation and management. This expertise changes in character as technology changes but it also changes because of the need to replace experts who leave.</p>
</blockquote>
<p>Furthermore, Cook writes that a complex system will always “contain practitioners and trainees with varying degrees of expertise”. Problems arise when knowledge isn’t spread equally in the team(s) responsible for the production stack.</p>
<p>I made the experience that <em>pair programming</em> is a very efficient way to share knowledge (yes, even in web operations). This is especially true when a legacy system is involved and your pairing partner happens to know more about it than he/she likes to admit…</p>
<h3 id="14-change-introduces-new-forms-of-failure">14. Change introduces new forms of failure</h3>
<p>As a matter of fact, even deliberate changes to web systems will often have <a href="/unintended-consequences">unintended negative consequences</a>. There’s a high rate of change and often a variety of processes leading to those changes. This makes it hard – if not impossible – to fully understand how all the bits and pieces resonate with each other under different conditions. Put another way, web systems are largely intractable, which is a major reason why outages are both unavoidable and unpredictable.</p>
<p>Cook adds to this what I consider one of the most useful insights I gained from his paper, worth quoting in length:</p>
<blockquote>
<p>The low rate of overt accidents in reliable systems may encourage changes, especially the use of new technology, to decrease the number of low consequence but high frequency failures. These changes maybe actually create opportunities for new, low frequency but high consequence failures. [Because they] occur at a low rate, multiple system changes may occur before an accident, making it hard to see the contribution of technology to the failure.</p>
</blockquote>
<h3 id="15-views-of-cause-limit-the-effectiveness-of-defenses-against-future-events">15. Views of “cause” limit the effectiveness of defenses against future events</h3>
<p>The statement that “post-accident remedies for human error are usually predicated on obstructing activities that can cause accidents” reminds me, more than anything else, of <a href="https://en.wikipedia.org/wiki/Security_theater">airport security theater</a>, which also does little to prevent further accidents.</p>
<p>Cook urges us to <em>not</em> increase the coupling of our web systems in a knee-jerk reaction to failure:</p>
<blockquote>
<p>Instead of increasing safety, post-accident remedies usually increase the coupling and complexity of the system. This increases the potential number of latent failures and also makes the detection and blocking of accident trajectories more difficult.</p>
</blockquote>
<h3 id="16-safety-is-a-characteristic-of-systems-and-not-of-their-components">16. Safety is a characteristic of systems and not of their components</h3>
<p>Chaos theory tells us that small causes – involving human action or not – can have large effects. <em>Everything is connected</em>. The paper says:</p>
<blockquote>
<p>Safety is an emergent property of systems; it does not reside in a person, device or department of an organization or system. Safety cannot be purchased or manufactured; it is not a feature that is separate from the other components of the system.</p>
</blockquote>
<p>Consider this example: <a href="/chaos-engineering-a-shift-in-mindset">Learning to embrace failure</a> – a prerequisite for reliability – requires a fundamental shift in the mindset of managers and employees, if not whole companies. We can’t build reliable web systems by merely improving the codebase.</p>
<h3 id="17-people-continuously-create-safety">17. People continuously create safety</h3>
<p>“Failure free operations”, as we learned today, “are the result of activities of people who work to keep the system within the boundaries of tolerable performance [on a moment by moment basis]”.</p>
<p>Most of these activities are well-known processes, probably documented in a runbook, such as reverting a bad deployment. Sometimes, however, it requires “novel combinations or de novo creations of new approaches” to repair a broken system. In my experience, the latter is in particular the case with <a href="http://blog.scalyr.com/2015/09/irreversible-failures-lessons-from-the-dynamodb-outage/">irreversible failures</a>, where you can’t simply undo the action that caused it.</p>
<h3 id="18-failure-free-operations-require-experience-with-failure">18. Failure free operations require experience with failure</h3>
<p>This last point is a topic near and dear to my heart.</p>
<p>When failure is the exception rather than the rule, <a href="/complacency-the-enemy-of-resilience">we risk becoming complacent</a>.</p>
<blockquote>
<p>Recognizing hazard and successfully manipulating system operations to remain inside the tolerable performance boundaries requires intimate contact with failure.</p>
</blockquote>
<p>For me, the perfect embodiment of this idea is <em>Chaos Engineering</em>, a discipline based on the realization that proactively triggering failures in a system – through intentional actions at the sharp end – is the best way to prepare for disaster.</p>
<p>If you want to learn more about Chaos Engineering, here are some links for you to check out:</p>
<ul>
<li><a href="/chaos-engineering-101">Chaos Engineering 101</a></li>
<li><a href="/blog/chaos-monkey-for-fun-and-profit/">Talk: Chaos Monkey for Fun and Profit</a></li>
<li><a href="/a-little-story-about-amazon-ecs-systemd-and-chaos-monkey">A Little Story about Amazon ECS, systemd, and Chaos Monkey</a></li>
</ul>
<hr />
<p>What better way to end this article than to leave you with Cook’s responses to <a href="https://medium.com/@MBWyHI1QkE66/thanks-mathias-glad-to-know-that-you-found-the-thoughts-provocative-47eed48af5db">part 1</a> and <a href="https://medium.com/@MBWyHI1QkE66/great-article-mathias-7e0714d4bbdf">part 2</a>, in which he provides additional context and links to his Velocity talks. Highly recommended for further study!</p>
How Complex Web Systems Fail - Part 12016-07-27T00:00:00+02:00https://sharpend.io/how-complex-web-systems-fail-part-1<p><img src="/assets/images/complex-1.jpg" alt="" /></p>
<p>There’s this one paper that keeps popping up on my radar. I think it’s about time I give it the attention it deserves. I’m talking about <a href="http://web.mit.edu/2.75/resources/random/How%20Complex%20Systems%20Fail.pdf">How Complex Systems Fail</a> by Richard Cook. This seminal paper, published in 2000, covers 18 sharp observations on the nature of failure in complex medical systems. The nice thing about these observations is that most of them hold true for complex systems in general, including our beloved <em>web systems</em>.</p>
<p>Distributed web-based systems are inherently complex. They’re composed of many moving parts – web servers, databases, load balancers, CDNs, routers, and a lot more – working together to form an intricate whole.</p>
<p>In this article, which is part 1 of 2, I’ll go through the first half of Cook’s observations, one by one, and try to translate them into the context of web systems. (<a href="/how-complex-web-systems-fail-part-2">In part 2</a>, I’ll cover the other half.)</p>
<h3 id="1-complex-systems-are-intrinsically-hazardous-systems">1. Complex systems are intrinsically hazardous systems</h3>
<p>This is certainly true for safety-critical systems in industries like medicine, transportation, or construction where errors can mean the difference between life and death. While most web systems fortunately don’t put our lives at risk, the general response to failures is the same: creating defense mechanisms against potential hazards inherent in those systems. Which brings us to the next point…</p>
<h3 id="2-complex-systems-are-heavily-and-successfully-defended-against-failure">2. Complex systems are heavily and successfully defended against failure</h3>
<p>We put countermeasures in place – backup systems, monitoring, DDoS protection, runbooks, GameDay exercises, etc. – because we dread the consequences of failure, such as service outages and data loss. These measures are supposed to “provide a series of shields that normally divert operations away from accidents”. And luckily, they’re successful most of the time.</p>
<h3 id="3-catastrophe-requires-multiple-failures--single-point-failures-are-not-enough">3. Catastrophe requires multiple failures – single point failures are not enough</h3>
<p>Cook writes:</p>
<blockquote>
<p>Overt catastrophic failure occurs when small, apparently innocuous failures join to create opportunity for a systemic accident. Each of these small failures is necessary to cause catastrophe but only the combination is sufficient to permit failure.</p>
</blockquote>
<p><em>Most</em> failure trajectories are successfully blocked by the aforementioned defenses or by the system operators themselves.</p>
<p>Later in this article, you’ll learn why there’s no such thing as a single root cause.</p>
<h3 id="4-complex-systems-contain-changing-mixtures-of-failures-latent-within-them">4. Complex systems contain changing mixtures of failures latent within them</h3>
<p>High complexity ensures there are multiple flaws – bugs – present at any given moment. Operators have to deal with ever-changing failures due to “changing technology, work organization, and efforts to eradicate failures”. Anyone who’s worked on a larger software project knows this is true. At some point, someone did something and it had an <a href="/unintended-consequences">unintended consequence</a>.</p>
<p>According to Cook, we don’t – and can’t – fix all latent bugs because of “economic cost but also because it is difficult before the fact to see how such failures might contribute to an accident”. We’re prone to think of these individual defects as “minor factors during operations”. However, as we just learned, several of these supposedly minor factors can lead to catastrophe.</p>
<h3 id="5-complex-systems-run-in-degraded-mode">5. Complex systems run in degraded mode</h3>
<p>A consequence of the preceding observation is that “complex systems run as broken systems”. Most of the time, they continue to work thanks to redundancies – database replicas, server auto scaling, etc. – and thanks to knowledgeable operators who fix problems as they arise.</p>
<p>But at some point <em>systems will fail</em>. It’s inevitable.</p>
<p>A postmortem might find that “the system has a history of prior ‘proto-accidents’ that nearly generated catastrophe” and that operators should have recognized the degradation in system performance before it was too late. However, that’s an oversimplified view. We need to realize, instead, that “system operations are dynamic, with components (organizational, human, technical) failing and being replaced continuously”. Attribution is not that simple, as you’ll see in a minute.</p>
<h3 id="6-catastrophe-is-always-just-around-the-corner">6. Catastrophe is always just around the corner</h3>
<blockquote>
<p>Disaster can occur at any time and in nearly any place. The potential for catastrophic outcome is a hallmark of complex systems. It is impossible to eliminate the potential for such catastrophic failure; the potential for such failure is always present by the system’s own nature.</p>
</blockquote>
<p>Just because there are no problems now doesn’t mean it’s going to stay that way. Sooner or later, any complex system will fail. That’s why operators should never get too comfortable.</p>
<p>As I wrote in the past, <a href="/complacency-the-enemy-of-resilience">complacency is the enemy of resilience</a>. The longer you wait for disaster to strike in production – merely hoping that everything will be okay – the less likely you are to handle emergencies well, both at a technical and organizational level.</p>
<h3 id="7-post-accident-attribution-to-a-root-cause-is-fundamentally-wrong">7. Post-accident attribution to a root cause is fundamentally wrong</h3>
<p>In complex systems, such as web systems, <a href="http://www.kitchensoap.com/2012/02/10/each-necessary-but-only-jointly-sufficient/">there is no root cause</a>. Instead, accidents require multiple contributors, each necessary but only jointly sufficient. In the words of Cook:</p>
<blockquote>
<p>Indeed, it is the linking of these causes together that creates the circumstances required for the accident. Thus, no isolation of the ‘root cause’ of an accident is possible.</p>
</blockquote>
<p>One of the reasons we tend to look for a single, simple cause of an outcome is because the failure is too complex to keep it in our head. Thus we oversimplify without really understanding the failure’s nature and then “blame specific, localized forces or events for outcomes”.</p>
<h3 id="8-hindsight-biases-post-accident-assessments-of-human-performance">8. Hindsight biases post-accident assessments of human performance</h3>
<p>The key point made here:</p>
<blockquote>
<p>Hindsight bias remains the primary obstacle to accident investigation, especially when expert human performance is involved.</p>
</blockquote>
<p>Wikipedia has a good explanation of <a href="https://en.wikipedia.org/wiki/Hindsight_bias">hindsight bias</a>:</p>
<blockquote>
<p>Hindsight bias, also known as the knew-it-all-along effect […] is the inclination, after an event has occurred, to see the event as having been predictable, despite there having been little or no objective basis for predicting it.</p>
</blockquote>
<p>Which tells us that it’s <em>impossible</em> to accurately assess human performance after an accident, e.g. when doing a postmortem. Still, many companies continue to blame people for mistakes when they should really blame – and fix – their <em>broken processes</em>.</p>
<h3 id="9-human-operators-have-dual-roles-as-producers-and-as-defenders-against-failure">9. Human operators have dual roles: as producers and as defenders against failure</h3>
<p>Operators actually have not one but <em>two</em> roles, each with its own demands. On the one hand, they operate the system so that it can do what it’s supposed to do. On the other hand, they defend the system against failures. According to Cook, this poses the following problem:</p>
<blockquote>
<p>Outsiders rarely acknowledge the duality of this role. In non-accident filled times, the production role is emphasized. After accidents, the defense against failure role is emphasized. At either time, the outsider’s view misapprehends the operator’s constant, simultaneous engagement with both roles.</p>
</blockquote>
<p>This duality reminds me, in a way, of today’s Site Reliability Engineers, who are responsible for ensuring that services are available and fast enough, and who also progress the software and systems behind those services. This duality is, in fact, at the heart of SRE. I’m glad our industry has started to embrace this idea.</p>
<hr />
<p>That’s the end of part 1. Of course, I encourage you to read the <a href="http://web.mit.edu/2.75/resources/random/How%20Complex%20Systems%20Fail.pdf">original treatise</a> to get the whole picture. I found a lot of value in it – and so might you.</p>
<p><a href="/how-complex-web-systems-fail-part-2">Continue with part 2 of this article</a>.</p>
Toil: A Word Every Engineer Should Know2016-07-13T00:00:00+02:00https://sharpend.io/toil-a-word-every-engineer-should-know<p><img src="/assets/images/toil.jpg" alt="" /></p>
<p><em>Toil</em>.</p>
<p>I’ve stumbled upon this word a few times already, mostly in the context of automation or system administration. I knew that it means something negative, something that should be avoided. But to be honest, it was only through writing this article – a useful technique for closing knowledge gaps, by the way – that I figured out what the word actually means when used in the world of engineering.</p>
<p>Here’s what I’ve learned.</p>
<h2 id="what-exactly-is-toil">What exactly is toil?</h2>
<p>For research, I’ve consulted two of my favorite technical books. <a href="http://the-cloud-book.com/">The Cloud book</a> mentions the term a couple of times in regard to automation and lists ways to assess and limit the amount of toil. <a href="https://landing.google.com/sre/book.html">The SRE book</a>, the primary source for this article, even has an entire chapter on toil and how to eliminate it.</p>
<p>The SRE book defines toil as follows:</p>
<blockquote>
<p>In SRE, we want to spend time on long-term engineering project work instead of operational work. Because the term operational work may be misinterpreted, we use a specific word: toil. […] Toil is the kind of work tied to running a production service that tends to be manual, repetitive, automatable, tactical, devoid of enduring value, and that scales linearly as a service grows.</p>
</blockquote>
<p>More precisely, the following work can be considered toil:</p>
<ul>
<li>Work such as manually running a script (even if the script itself automates some task)</li>
<li>Work that is performed over and over again</li>
<li>Work that could just as well be accomplished by a machine (human judgment isn’t essential)</li>
<li>Work that is interrupt-driven and reactive, like pager alerts, rather than strategy-driven and proactive</li>
<li>Work that does not <em>permanently</em> improve your service once completed</li>
<li>Work that scales up linearly with service size, traffic volume, or user count</li>
</ul>
<h2 id="why-is-toil-a-bad-thing">Why is toil a bad thing?</h2>
<p>Toil is not a bad thing per se. There’ll always be some amount of toil – unavoidable grunge work – that you need to take care of as an engineer. You might even look forward to these quick wins from time to time. That’s fine. There’s a problem, however, if toil makes up the <em>majority</em> of your work.</p>
<p>Here are some of the reasons why too much toil is harmful, again taken from the SRE book:</p>
<ul>
<li>Too much toil leads to burnout, boredom, and discontent.</li>
<li>Manual work and firefighting will prevail at the expense of shipping new features.</li>
<li>Too much toil creates confusion about what SRE actually entails.</li>
<li>Other (development) teams may start expecting SREs to take on even more toil.</li>
<li>Current or future teammates are more likely to look for another job, esp. if they were promised project work.</li>
<li>Your career will stagnate if you spend too little time on <em>long-term engineering projects</em>.</li>
</ul>
<p>I think the last aspect is particularly important. But what exactly qualifies as engineering work?</p>
<h2 id="engineering-work-defined">Engineering work defined</h2>
<blockquote>
<p>Engineering work is novel and intrinsically requires human judgment. It produces a permanent improvement in your service, and is guided by a strategy. It is frequently creative and innovative, taking a design-driven approach to solving a problem – the more generalized, the better. Engineering work helps your team or the SRE organization handle a larger service, or more services, with the same level of staffing.</p>
</blockquote>
<p>Examples include:</p>
<ul>
<li>Creating automation scripts, tools, or frameworks</li>
<li>Adding service features for scalability and reliability</li>
<li>Modifying infrastructure code to make it more robust</li>
<li>Configuring and tuning production systems (servers, load balancers, etc.)</li>
<li>Setting up monitoring and alerting</li>
<li>Writing documentation that has long-term value</li>
<li>Consulting on architecture, design and putting systems into production</li>
</ul>
<h2 id="the-5050-rule">The 50/50 rule</h2>
<p>Without a doubt, knowing what work falls into which category – toil or long-term engineering work – is useful. Another thing you need to know is how to split your valuable time between the two.</p>
<p>Fortunately, the SRE book has a pragmatic answer for this too:</p>
<blockquote>
<p>Our SRE organization has an advertised goal of keeping operational work (i.e., toil) below 50% of each SRE’s time. At least 50% of each SRE’s time should be spent on engineering project work that will either reduce future toil or add service features. Feature development typically focuses on improving reliability, performance, or utilization, which often reduces toil as a second-order effect.</p>
</blockquote>
<p>From now on, I’m going to strive for spending, on average, at least 50% of my time on engineering work. Furthermore, it’s time for me to stop using the phrase “monkey work” because, as we learned today, there exists a much better, more empathetic word.</p>
If it hurts, do it more often2016-06-30T00:00:00+02:00https://sharpend.io/if-it-hurts-do-it-more-often<p><img src="/assets/images/lift.jpg" alt="" /></p>
<p>There’s a good chance that regular exercise will help you become a healthier person. The more often you go to the gym and lift weights (or whatever you like to do), the more strength you will gain, and as a result, the easier it becomes to repeat the same exercise. This is possible because your body will slowly but gradually adapt to the stress of exercise.</p>
<p>While that’s obviously a simplified explanation – there’s a lot more to be said about adaptation and exercising the right way without getting injured – this is generally true:</p>
<p>Practice something long enough and you will get better at it. Or putting it in a more catchy way: <strong>If it hurts, do it more often.</strong> Eventually, the pain will fade away.</p>
<p>The great thing about this principle is that it’s not limited to sports – you can apply it to any useful activity that can be done frequently:</p>
<ul>
<li>Do you consider yourself to be a mediocre speaker? Keep giving presentations at local user groups until you feel more comfortable.</li>
<li>Do you have trouble writing blog posts? Make it a habit to <a href="/blog/write-every-day">write 250 words a day</a> and the pain will vanish (mostly).</li>
<li>A different kind of example: Entrepreneurs are supposed to wear many hats. By spending time in customer service, they can learn what it takes to provide great support – and only then consider hiring for the position.</li>
</ul>
<p>Now let’s see how this idea applies to software development, using continuous integration as an example.</p>
<h2 id="the-pain-of-continuous-integration">The pain of continuous integration</h2>
<p>In his post, <a href="http://martinfowler.com/bliki/FrequencyReducesDifficulty.html">Frequency Reduces Difficulty</a>, Martin Fowler writes:</p>
<blockquote>
<p>Most programmers learn early on that integrating their work with others is a frustrating and painful experience. The natural human response, therefore, is to put off doing it for as long as possible.</p>
</blockquote>
<p>He continues:</p>
<blockquote>
<p>[…] if you do it more frequently, you can drastically reduce the pain. And this is what happens with Continuous Integration – by integrating every day, the pain of integration almost vanishes. It did hurt, so you did it more often, and now it no longer hurts.</p>
</blockquote>
<p>But why is doing <em>painful</em> things over and over a good idea?</p>
<p>According to Fowler, there are three main reasons, all of them manifested in agile thinking:</p>
<ul>
<li>
<p><strong>Decomposition.</strong> Before you can execute tasks more frequently, you need to decompose large tasks into smaller chunks to make them easier to handle. Today we mainly talk about decomposing applications into microservices. However, breaking complex systems down into smaller classes or objects has been a best practice of object-oriented programming for decades.</p>
</li>
<li>
<p><strong>Feedback.</strong> In software development, <a href="/fast-feedback-is-everything">fast feedback is important</a> in order to adjust more quickly to changes. Fast feedback involves reducing the time between modifying code and getting test results. Being able to do something more frequently, like integrating code changes, leads to faster feedback.</p>
</li>
<li>
<p><strong>Automation.</strong> As I touched on earlier, practice – together with reflection – help you improve any skill. Having to redo a time-consuming task many times makes you want to automate it because you know that <em>reliable</em> automation increases speed and reduces errors. After feeling the pain for some time, you’re in a better position to automate the task based on newly acquired knowledge.</p>
</li>
</ul>
<p><strong>So pain doesn’t necessarily have to be a bad thing. It can lead to better systems that are decomposed, automated, and easier to change and reason about.</strong></p>
<h2 id="our-staging-environment">Our staging environment</h2>
<p>We, the members of Jimdo’s Werkzeugschmiede team, also like to feel the pain before doing something about it.</p>
<p>For example, before investing time in setting up a proper staging environment, we had experienced how difficult it was to test changes prior to deploying them to production. Later, when staging was in place, we learned the hard way that our automation wasn’t as reliable as we thought. We ignored the problem for a while – again feeling the pain – until we finally decided to rebuild staging from scratch once a week, thereby ironing out any flaws in our automation.</p>
<p>There are still other things that hurt us in different ways. Let’s do those things more often and see where it leads us…</p>
A Little Story about Amazon ECS, systemd, and Chaos Monkey2016-06-13T00:00:00+02:00https://sharpend.io/a-little-story-about-amazon-ecs-systemd-and-chaos-monkey<p>Today I want to share a little story. The story is about the challenge of making one component of our core infrastructure more resilient to failures.</p>
<p>Before jumping to the meat of it, however, it will be helpful to have an understanding of what the infrastructure stack in question does and how it works.</p>
<h2 id="a-quick-look-at-wonderland">A quick look at Wonderland</h2>
<p>I’m currently working as an Infrastructure Toolsmith in Jimdo’s Werkzeugschmiede team. In brief, our goal is to provide a platform (PaaS) where all Jimdo developers can deploy and run their Dockerized applications with as little friction as possible.</p>
<p>Our platform, which we internally call <em>Wonderland</em>, utilizes the Amazon EC2 Container Service (ECS). ECS is essentially a cluster scheduler that maps a set of work (batch jobs or long-running services) to a set of resources (EC2 instances). To connect instances to an ECS cluster, one has to install and run a daemon called <em>ecs-agent</em> on them. ecs-agent will then start Docker containers on behalf of ECS. Wonderland, in turn, talks to the ECS API whenever a user triggers a deployment.</p>
<p>There are actually more moving pieces involved, but that’s the gist of it.</p>
<h2 id="what-can-go-wrong">What can go wrong</h2>
<p>For our setup to work, it’s important that the following conditions are met on each cluster instance:</p>
<ul>
<li>ecs-agent is running properly to get instructions from ECS.</li>
<li>Docker daemon is running properly to start containers on behalf of ecs-agent.</li>
<li>/var/lib/docker is backed by a functioning Amazon EBS volume with enough storage for Docker.</li>
</ul>
<p>According to Murphy’s law, <strong>anything that can go wrong will go wrong</strong>. Here are some of the problems we had to deal with in the past:</p>
<ul>
<li>Deploying a newer (and broken) version of ecs-agent that would disconnect from ECS after a while. We did a rollback. These days, we do more testing before updating ecs-agent in production.</li>
<li>Docker daemon becoming unresponsive. After lots of debugging, we found out this was due to Datadog agent collecting disk usage metrics per container. After disabling the corresponding “container_size” metric the problem was gone.</li>
<li>EBS volume running out of inodes (“no space left on device”), which was caused by Docker containers with many small files in them. The fix was to switch the underlying filesystem from EXT4 to XFS.</li>
<li>Formatting of EBS volume failed with “mkfs.xfs: /dev/xvdb appears to contain an existing filesystem (xfs)”. Without the mounted device, Docker quickly filled up the small root filesystem. Only happened once, though. We ignored it.</li>
</ul>
<h2 id="self-healing-infrastructure">Self-healing infrastructure</h2>
<p>Of course, we want our infrastructure to be resilient to failures. If an EC2 instance becomes unhealthy – if it’s unable to run Docker containers for whatever reason – it should be replaced without any manual intervention.</p>
<p>To achieve this, all of our cluster instances are managed by an EC2 auto scaling group. The associated load balancer is configured to send an HTTP request to each instance to perform a health check (the specific endpoint is <code class="language-plaintext highlighter-rouge">:51678/v1/metadata</code>). If ecs-agent does not respond with “200 OK” for some time, the instance will be terminated and a new one will be started immediately.</p>
<p>That’s the theory. In practice, monitoring ecs-agent alone is not enough, as the used endpoint will happily report that everything is fine even if, for example, /var/lib/docker isn’t writable…</p>
<p>Now what?</p>
<h2 id="systemd-to-the-rescue">systemd to the rescue</h2>
<p>For the cluster instances, we decided to use CoreOS, a container-focused Linux distribution optimized for large-scale deployments. Like most modern Linux distributions, CoreOS uses systemd as its init system.</p>
<p>These are the systemd unit files we use that are of interest:</p>
<ul>
<li>format-ephemeral.service – formats the EBS volume using mkfs.xfs</li>
<li>var-lib-docker.mount – mounts the EBS volume to /var/lib/docker</li>
<li>docker.service – starts the Docker daemon (this unit ships with CoreOS)</li>
<li>ecs-agent.service – starts ecs-agent</li>
</ul>
<p>From the beginning, we had requirement dependencies in place (via <a href="https://www.freedesktop.org/software/systemd/man/systemd.unit.html#Requires=">Requires=</a>) to ensure units are started and stopped at the right time. For example, var-lib-docker.mount requires format-ephemeral.service, and ecs-agent.service requires docker.service to work.</p>
<p>We want to take advantage of the fact that <em>systemd will stop ecs-agent</em> if any of its dependencies fails, which would cause the unhealthy instance to be replaced. Self-healing infrastructure for the win!</p>
<h2 id="the-missing-link">The missing link</h2>
<p>The attentive reader will already have noticed that – until recently – we had been missing one important dependency in the chain: ecs-agent, which provides our health check endpoint, <em>absolutely</em> requires the EBS volume to be formatted and read-write mounted to /var/lib/docker. Without the volume, the cluster instance can’t work reliably.</p>
<p>I tried to remedy this fact by adding <code class="language-plaintext highlighter-rouge">Requires=var-lib-docker.mount</code> to ecs-agent.service. Some testing showed, however, that the agent won’t be stopped if the mount point is unmounted at runtime. Luckily, one of my colleagues suggested to give <a href="https://www.freedesktop.org/software/systemd/man/systemd.unit.html#BindsTo=">BindsTo=</a> a try instead:</p>
<blockquote>
<p>Configures requirement dependencies, very similar in style to <code class="language-plaintext highlighter-rouge">Requires=</code>, however in addition to this behavior, it also declares that this unit is stopped when any of the units listed suddenly disappears. Units can suddenly, unexpectedly disappear if a service terminates on its own choice, a device is unplugged or a mount point unmounted without involvement of systemd.</p>
</blockquote>
<p>Which turned out to be what we needed. With <code class="language-plaintext highlighter-rouge">BindsTo=</code> in place, I executed <code class="language-plaintext highlighter-rouge">umount /var/lib/docker</code> and <code class="language-plaintext highlighter-rouge">systemctl stop var-lib-docker.mount</code>, and in both cases ecs-agent was stopped by systemd – and the instance was terminated!</p>
<p>So all is well. Or is it?</p>
<h2 id="chaos-monkey-and-xfs">Chaos Monkey and XFS</h2>
<p>To be honest, neither unmounting the filesystem nor stopping the mount unit are good ways to simulate a failure of EBS. I knew how to do better, though.</p>
<p><a href="/chaos-monkey-for-fun-and-profit">As a big fan of Chaos Monkey</a>, I couldn’t resist using it for a quick chaos experiment: What would happen if, instead of unmounting the EBS volume gracefully, it was <em>forcefully detached</em> at runtime? I assumed that the result would be the same: systemd would stop ecs-agent.</p>
<p>To validate my assumption, <a href="https://github.com/mlafeldt/chaosmonkey">I used a little command-line tool</a> I’ve developed to trigger chaos events via the Chaos Monkey REST API:</p>
<figure class="highlight"><pre><code class="language-sh" data-lang="sh"><span class="nv">$ </span>chaosmonkey <span class="nt">-endpoint</span> <span class="s2">"</span><span class="nv">$WONDERLAND_ENDPOINT</span><span class="s2">"</span> <span class="se">\</span>
<span class="nt">-username</span> <span class="s2">"</span><span class="nv">$WONDERLAND_USER</span><span class="s2">"</span> <span class="nt">-password</span> <span class="s2">"</span><span class="nv">$WONDERLAND_PASS</span><span class="s2">"</span> <span class="se">\</span>
<span class="nt">-group</span> wonderland-crims-CrimsAutoScalingGroup <span class="se">\</span>
<span class="nt">-strategy</span> DetachVolumes</code></pre></figure>
<p>The “DetachVolumes” chaos strategy will force-detach all EBS volumes from a randomly selected EC2 instance of the given auto scaling group. As a result, EBS disk I/O will fail.</p>
<p>The result was surprising, at least to me.</p>
<p><a href="https://github.com/systemd/systemd/issues/3468">You can read up the details here</a>. In a nutshell, systemd did <em>not</em> terminate ecs-agent because of XFS. If an error was detected, the Linux XFS driver – by design – won’t unmount the filesystem in order to keep it consistent (reading or writing data will result in I/O errors). Hence, systemd correctly reported that the device was still mounted, leaving us with a broken system that won’t be able to start new Docker containers…</p>
<h2 id="conclusion">Conclusion</h2>
<p>Unfortunately, we cannot rely on our current health check to correctly detect EBS failures. To make up for this, we probably need to provide our own monitoring daemon for more sophisticated checks. This is something we wanted to avoid because, as the saying goes, no code is better than no code.</p>
<p>I myself have learned a lot about systemd, EBS, and XFS. Even though our infrastructure isn’t perfect yet, we’re more confident in our system now than we were prior to the experiments. It’s not the first time <a href="/chaos-engineering-101">Chaos Engineering</a> has proved our assumptions wrong.</p>
<p><strong>Still, there’s much to learn from failure.</strong></p>
Simplicity: A Prerequisite for Reliability2016-05-31T00:00:00+02:00https://sharpend.io/simplicity-a-prerequisite-for-reliability<p><img src="/assets/images/simplicity.jpg" alt="" /></p>
<p>I started reading the <a href="https://landing.google.com/sre/book.html">Site Reliability Engineering book</a>, commonly referred to as “the SRE book”. In this collection of essays, members of Google’s SRE team write about how they run their production systems. The book contains a short chapter on a topic near and dear to my engineering heart: <strong>simplicity</strong> and the role it plays in SRE. In this article, I’m going to summarize the lessons from that chapter and add some thoughts of my own.</p>
<h2 id="stability-vs-agility">Stability vs. agility</h2>
<p>SRE is about keeping agility and stability of production systems in balance. Running a reliable service is easy – because it’s frozen. But: running an <em>agile</em> reliable service is hard. The challenge is to change an application that is running correctly, while keeping the service running correctly.</p>
<p>Reliable processes can increase developer agility. Reliable production rollouts, for example, make it easier to link changes to bugs. This in turn allows developers to focus on more important things, such as functionality and performance of their software.</p>
<p>This separation of concerns is, in fact, one of the reasons why cluster managers like Kubernetes exist. Developers can talk to the Kubernetes API to deploy their application containers in a simple and reliable way without having to worry about cluster nodes, host OS/kernel, or underlying hardware.</p>
<h2 id="accidental-vs-essential-complexity">Accidental vs. essential complexity</h2>
<p>Software should behave predictably and accomplish its goals without too many surprises (that is, outages in production). The number of surprises directly correlates with the amount of unnecessary complexity found in a project. It’s therefore crucial to think about accidental complexity and essential complexity:</p>
<blockquote>
<p>Accidental complexity relates to problems which engineers create and can fix, [whereas] essential complexity is caused by the problem to be solved, and nothing can remove it<br />
– Fred Brooks in his seminal “No Silver Bullet” essay</p>
</blockquote>
<p>SRE teams should push back when accidental complexity is introduced into the systems they’re in charge of. Besides, they should <em>continue</em> to eliminate complexity over time.</p>
<p>As a matter of fact, it can be hard to know what is essential complexity and what is accidental complexity. From my experience, you can successfully avoid unnecessary complexity by:</p>
<ul>
<li>Constantly asking yourself what it is you want to accomplish</li>
<li>Thinking deeply about the problem domain</li>
<li>Saying “no” by default</li>
<li>Outsourcing work to service providers, if possible</li>
</ul>
<p>(I’m aware that these points need more elaboration, most likely in the form of another article.)</p>
<h2 id="code-is-a-liability">Code is a liability</h2>
<p>It’s poor practice to comment out unused code, or worse, to gate it with a feature flag. Code that has no purpose is a major source of distraction and confusion. Today’s version control systems make it easy to revert any changes; there’s no reason not to remove dead code and other bloat. <a href="https://gettingreal.37signals.com/ch10_Less_Software.php">Less code means less complexity</a>, which means less bugs, which means less unexpected outcomes in production. Deleting many – sometimes hundreds or thousands – lines of code is indeed very satisfying.</p>
<p>At the same time, we should think twice before adding new features. Once again, this comes down to essential vs. accidental complexity and saying “no” to many things in order to focus on the core problem.</p>
<h2 id="minimal-apis">Minimal APIs</h2>
<p>Writing clear, minimal APIs is key to managing simplicity in software systems. Smaller APIs with fewer methods and arguments are not only easier to understand and test, they also allow us to put more effort into comprehending the actual problem we set out to solve.</p>
<p>As you can see, there’s a pattern involved here: less is more.</p>
<h2 id="modularity-and-decoupling">Modularity and decoupling</h2>
<p>Many concepts of object-oriented programming (OOP) also apply to the design of distributed systems. For instance, both involve breaking problems up into small, manageable components. No surprise then that both benefit from loose coupling – the ability to update parts of a system in isolation – as an effective method for increasing developer agility and system stability. A decoupled system, in particular, reduces the probability of <a href="/unintended-consequences">unintended consequences</a>.</p>
<p>The idea of modularity also extends to APIs. API versioning allows developers to upgrade individual components to a newer version in a controlled way, rather than forcing all teams to upgrade in “big bang” fashion. What appears to be an extra burden at first, actually allows introducing new features and deprecating old ones in a safe manner.</p>
<p>Similar to how all code should have a purpose, any part of a distributed system – microservices, binaries, etc. – should be responsible for solving one particular well-defined problem. Those parts are connected by clearly defined interfaces (API, CLI, etc). This, in fact, is the <a href="https://en.wikipedia.org/wiki/Unix_philosophy">Unix philosophy</a> at work.</p>
<h2 id="small-releases">Small releases</h2>
<p>Nobody likes to review pull requests that are too long. It’s hard to measure the impact of many code changes at a time, even more so if the changes are unrelated. The same is true for releases. It’s difficult to understand the impact (e.g. on performance) when deploying dozens of unrelated changes to a system at the same time.</p>
<p>Simple releases performed in smaller batches of changes are easier to measure – and to revert, if necessary.</p>
<h2 id="conclusion">Conclusion</h2>
<p>I want to end this article, quite appropriately, with one of my favorite quotes:</p>
<blockquote>
<p>Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it.<br />
– Kernighan and Plauger in “The Elements of Programming Style”</p>
</blockquote>
<p>The more complex a system, the more difficult it is to build a mental model of the system, and the harder it becomes to operate and debug it. As Edsger W. Dijkstra put it, “Simplicity is prerequisite for reliability”. It’s important to put a lot of thought into simplifying our designs while still providing the required functionality. Systems tend to become more complex over time – the earlier you start simplifying as outlined here, the better off you will be.</p>
Fast Feedback Is Everything2016-05-19T00:00:00+02:00https://sharpend.io/fast-feedback-is-everything<blockquote>
<p>As an engineer, you should constantly work to make your feedback loops shorter in time and/or wider in scope. – <a href="https://twitter.com/KentBeck/status/531964254946328576">@KentBeck</a></p>
</blockquote>
<p>As software engineers, we spend a lot of our time writing code. Whether we are implementing new features, fixing nasty bugs, or doing boring maintenance work, there is always some code we either create from scratch or try to modify for the better. When developing code, we need certainty that our changes work as intended. That’s why we write tests after (or before) the fact. As a result, a huge part of our daily work inevitably comes down to these steps:</p>
<ol>
<li>Write some code.</li>
<li>Write a test that defines the desired behavior.</li>
<li>Run the test to see if it’s successful.</li>
<li>Go back to 1. or 2. if the test fails. Rinse, repeat.</li>
</ol>
<p>(For the sake of this article, let’s ignore the tedious test-first vs. test-last vs. TDD debate.)</p>
<p>In order to write an actual program, you have to run through these steps – this loop – again and again. But here’s the thing: constantly jumping back and forth between programming and testing comes at a price. It doesn’t just slow you down in terms of time spent; the involved <em>context switching</em> also drains your mental energy, which might ultimately destroy your productivity.</p>
<p>For this reason, I believe that the following statement is so important – and I’m not tired of repeating it whenever I get the chance:</p>
<p><strong>When it comes to programming and testing, fast feedback is everything.</strong></p>
<p>Fast feedback involves reducing the time between changing code (or tests) and getting test results to a minimum. The faster this feedback loop, the more productive you will be.</p>
<p>There are a couple things you can do to shorten the feedback loop. Certainly the first technique coming to mind is isolated testing, which involves eliminating (slow) external dependencies like databases. Besides trying to implement faster tests, however, you can also optimize the way you <em>run</em> them.</p>
<h2 id="running-tests-the-fast-way">Running tests the fast way</h2>
<p>At first glance, <em>how</em> you run tests might not seem to have a big impact on the feedback loop. If a test takes a minute to finish, does it really matter if we can shave off a second or two by tweaking the running step?</p>
<p><em>Yes, it does.</em> Seconds add up over time. Each additional step requires a little more brain power and incurs a significant context-switching cost.</p>
<p>I don’t like wasting my time with work that can easily be avoided. If there’s a way to minimize the cost of context switching, I’m more than happy to add it to my toolbox. By following the following three steps, I’ve managed to run tests faster and, more importantly, become more productive as a result.</p>
<ol>
<li>
<p><strong>Figure out how to execute individual tests.</strong> During development, don’t run the entire test suite each time you change a bit of code. Aside from the fact that running all tests is often very slow, it’s always better (and faster!) to first get feedback on local code changes before integrating with other code. Reducing the scope by testing a small subset of code in isolation is not only faster, it also helps you find bugs, and it’s a must-have for TDD. (It goes without saying that you or your continuous integration system should run all the tests at some point.)</p>
</li>
<li>
<p><strong>Write a test runner.</strong> This is optional and depends on your test framework/setup. For example, RSpec – one of the best frameworks for testing in Ruby – already allows you to execute a specific test file or even a single test case in that file. Unfortunately, it’s not always that easy. Sometimes you need to execute additional setup/teardown tasks, other times running tests on a package level may be the best you can do. That’s where a test runner comes in handy. In its most basic form, a test runner is a shell script that takes a single argument – the filename of the test you’re currently working on – and does everything required to run the test. I usually store this script as <code class="language-plaintext highlighter-rouge">script/test</code> in every project I need it.</p>
</li>
<li>
<p><strong>Run tests using a keyboard shortcut.</strong> For fast feedback, it’s important to not leave your editor while hacking on code. Instead, configure your editor of choice to execute tests when pressing a combination of keys on your keyboard. At a minimum, set up a shortcut to run the test currently open in your editor by passing its filename directly to the respective testing tool – or a custom test runner. It’s also useful to have a shortcut for running the test case under the cursor, which will further narrow the focus of your testing.</p>
</li>
</ol>
<h2 id="examples">Examples</h2>
<p>Let me give you three practical examples. All of them are somehow related to infrastructure automation, both because it’s an area where rapid feedback matters all the more and because it’s what I do for a living. You will see that I’m a fan of Vim, but it should be straightforward to achieve the same with other editors as well. Here we go:</p>
<ul>
<li>
<p>rspec-puppet is a test framework that allows to write RSpec tests for Puppet code. When I started working at Jimdo in 2013, it wasn’t possible to run individual tests by simply pointing RSpec at a test file in our codebase. One reason is the unusual way test fixtures are handled in the Puppet world. To remedy this, I wrote a <a href="https://gist.github.com/mlafeldt/dc695074416bec4cb7a3">test runner script</a>. Together with <a href="https://github.com/Jimdo/vim-spec-runner">vim-spec-runner</a>, a Vim plugin that automatically sets up keyboard shortcuts for running tests, we had everything in place for testing our Puppet code at the whim of a keystroke.</p>
</li>
<li>
<p>I primarily developed <a href="https://github.com/mlafeldt/chef-runner">chef-runner</a> for use with Vim. Instead of jumping back and forth between editing a Chef recipe and running the painfully slow <code class="language-plaintext highlighter-rouge">vagrant provision</code> command, I wanted to be able to change code and get immediate feedback without having to leave the editor. chef-runner’s ability to rapidly provision a machine with just a single Chef recipe – the file currently open in Vim – made this possible. There’s no Vim plugin; the setup is as simple as <a href="https://github.com/mlafeldt/chef-runner/wiki/Vim">sticking a one-liner</a> in your Vim configuration.</p>
</li>
<li>
<p>chef-runner used to be a 100-line shell script before I decided to rewrite it in Go. Go comes with first-class testing support. The <code class="language-plaintext highlighter-rouge">go test</code> command is used to run tests (<code class="language-plaintext highlighter-rouge">_test.go</code> files) and report test results. However, the tool itself can only run tests for one or more packages based on their import paths; it cannot handle arbitrary <code class="language-plaintext highlighter-rouge">_test.go</code> files. These days, I use the <code class="language-plaintext highlighter-rouge">:GoTest</code> command from the excellent <a href="https://github.com/fatih/vim-go">vim-go</a> plugin to execute package tests for a specific source file.</p>
</li>
</ul>
<h2 id="wrapping-up">Wrapping up</h2>
<p>Fast feedback plays an important role in software development. Optimizing the way we run tests is one effective method to shorten the feedback loop and get things done.</p>
<p>Sometimes all it takes is a tiny shell script.</p>
<p><em>Acknowledgment: The ideas presented in this article were heavily inspired by the excellent <a href="https://www.destroyallsoftware.com/">Destroy All Software screencasts</a> by Gary Bernhardt.</em></p>
Chaos Monkey for Fun and Profit2016-05-04T00:00:00+02:00https://sharpend.io/chaos-monkey-for-fun-and-profit<p><img src="/assets/images/chaos-monkey.jpg" alt="" /></p>
<p>This article will pick up where <a href="/chaos-engineering-101">Chaos Engineering 101</a> left off and cover a slightly more advanced principle of Chaos Engineering: <strong>automating chaos experiments</strong>.</p>
<p>The core idea of Chaos Engineering, as we recall, is to inject failures proactively in a controlled manner in order to gain confidence in our systems. Chaos Engineering enables us to verify that things behave as we expect – and to fix them if they don’t.</p>
<p>In Chaos Engineering 101, I argued that you don’t need to automate chaos experiments when you’re just getting started. I still think that manual testing, which can be as simple as terminating a process with the <code class="language-plaintext highlighter-rouge">kill</code> command, is the easiest way to get familiar with the concept of fault injection and to gradually establish <a href="/chaos-engineering-a-shift-in-mindset">the right mindset</a>. At the end of the article, I explained how Jimdo runs GameDay events, which are typically based on manual fault injection as well.</p>
<h2 id="the-next-level-of-chaos">The next level of chaos</h2>
<p><a href="https://principlesofchaos.org/">The Principles of Chaos Engineering</a>, as formulated by Netflix, currently list four advanced principles of chaos. The document says:</p>
<blockquote>
<p>The [advanced] principles describe an ideal application of Chaos Engineering […] The degree to which these principles are pursued strongly correlates to the confidence we can have in a distributed system at scale.</p>
</blockquote>
<p>The one principle we’re interested in today is described as follows:</p>
<blockquote>
<p>Running experiments manually is labor-intensive and ultimately unsustainable. Automate experiments and run them continuously. Chaos Engineering builds automation into the system to drive both orchestration and analysis.</p>
</blockquote>
<p><strong>In other words, the Principles suggest automating experiments (that used to be manual) to run continuously in order to further increase confidence in our systems.</strong></p>
<p>Fortunately, Netflix does not only tell us what to do; they also gave us a mighty tool for putting theory into practice: <strong>Chaos Monkey</strong>.</p>
<h2 id="chaos-monkey">Chaos Monkey</h2>
<p>Netflix went the extra mile and built several autonomous agents, so-called “monkeys”, for injecting failures and creating different kinds of outages in an automated manner. Latency Monkey, for example, induces artificial delays in API calls to simulate service degradation, whereas Chaos Gorilla is programmed to take down an entire AWS availability zone. Together these monkeys form the <em>Simian Army</em>.</p>
<p>Chaos Monkey is the most famous member of Netflix’s Simian Army. In fact, it’s the first, and to this date only, monkey of its kind that is publicly available. Broadly speaking, Chaos Monkey randomly terminates EC2 instances in AWS. Here is a more thorough description from <a href="http://techblog.netflix.com/2011/07/netflix-simian-army.html">the Netflix blog</a>:</p>
<blockquote>
<p>[Chaos Monkey is] a tool that randomly disables our production instances to make sure we can survive this common type of failure without any customer impact. The name comes from the idea of unleashing a wild monkey with a weapon in your data center (or cloud region) to randomly shoot down instances and chew through cables – all the while we continue serving our customers without interruption.</p>
</blockquote>
<p>The post continues:</p>
<blockquote>
<p>By running Chaos Monkey in the middle of a business day, in a carefully monitored environment with engineers standing by to address any problems, we can still learn the lessons about the weaknesses of our system, and build automatic recovery mechanisms to deal with them. So next time an instance fails at 3 am on a Sunday, we won’t even notice.</p>
</blockquote>
<p>Next, I’ll show you how to run your very own Chaos Monkey.</p>
<h2 id="the-simian-army---docker-edition">The Simian Army - Docker Edition</h2>
<p>I spent a couple hours and dockerized the Simian Army, a Java application with dozens of settings, to make it as simple as possible to use Chaos Monkey. The result is <strong><a href="https://github.com/mlafeldt/docker-simianarmy">a highly configurable Docker image</a></strong> which, I hope, provides a sound basis for automating chaos experiments.</p>
<p>As an example, this command will start a Docker container running the Simian Army and instruct Chaos Monkey to consider all auto scaling groups (ASGs) in the given AWS account for termination:</p>
<figure class="highlight"><pre><code class="language-sh" data-lang="sh">docker run <span class="nt">-it</span> <span class="nt">--rm</span> <span class="se">\</span>
<span class="nt">-e</span> <span class="nv">SIMIANARMY_CLIENT_AWS_ACCOUNTKEY</span><span class="o">=</span><span class="nv">$AWS_ACCESS_KEY_ID</span> <span class="se">\</span>
<span class="nt">-e</span> <span class="nv">SIMIANARMY_CLIENT_AWS_SECRETKEY</span><span class="o">=</span><span class="nv">$AWS_SECRET_ACCESS_KEY</span> <span class="se">\</span>
<span class="nt">-e</span> <span class="nv">SIMIANARMY_CLIENT_AWS_REGION</span><span class="o">=</span><span class="nv">$AWS_REGION</span> <span class="se">\</span>
<span class="nt">-e</span> <span class="nv">SIMIANARMY_CALENDAR_ISMONKEYTIME</span><span class="o">=</span><span class="nb">true</span> <span class="se">\</span>
<span class="nt">-e</span> <span class="nv">SIMIANARMY_CHAOS_ASG_ENABLED</span><span class="o">=</span><span class="nb">true</span> <span class="se">\</span>
mlafeldt/simianarmy</code></pre></figure>
<p>This example is safe to run as Chaos Monkey will operate in dry-run mode by default. It’s a good way for getting a feeling of the application without taking a risk.</p>
<p>The second example is more realistic and could very well be your first chaos experiment to run continuously. This time, Chaos Monkey will randomly terminate instances of the auto scaling groups tagged with a specific key-value pair:</p>
<figure class="highlight"><pre><code class="language-sh" data-lang="sh">docker run <span class="nt">-it</span> <span class="nt">--rm</span> <span class="se">\</span>
<span class="nt">-e</span> <span class="nv">SIMIANARMY_CLIENT_AWS_ACCOUNTKEY</span><span class="o">=</span><span class="nv">$AWS_ACCESS_KEY_ID</span> <span class="se">\</span>
<span class="nt">-e</span> <span class="nv">SIMIANARMY_CLIENT_AWS_SECRETKEY</span><span class="o">=</span><span class="nv">$AWS_SECRET_ACCESS_KEY</span> <span class="se">\</span>
<span class="nt">-e</span> <span class="nv">SIMIANARMY_CLIENT_AWS_REGION</span><span class="o">=</span><span class="nv">$AWS_REGION</span> <span class="se">\</span>
<span class="nt">-e</span> <span class="nv">SIMIANARMY_CALENDAR_ISMONKEYTIME</span><span class="o">=</span><span class="nb">true</span> <span class="se">\</span>
<span class="nt">-e</span> <span class="nv">SIMIANARMY_CHAOS_ASG_ENABLED</span><span class="o">=</span><span class="nb">true</span> <span class="se">\</span>
<span class="nt">-e</span> <span class="nv">SIMIANARMY_CHAOS_ASGTAG_KEY</span><span class="o">=</span>chaos_monkey <span class="se">\</span>
<span class="nt">-e</span> <span class="nv">SIMIANARMY_CHAOS_ASGTAG_VALUE</span><span class="o">=</span><span class="nb">true</span> <span class="se">\</span>
<span class="nt">-e</span> <span class="nv">SIMIANARMY_CHAOS_LEASHED</span><span class="o">=</span><span class="nb">false</span> <span class="se">\</span>
mlafeldt/simianarmy</code></pre></figure>
<p>Note that this command will actually <em>unleash</em> the monkey. But don’t worry: you still need to tag your ASGs accordingly for any instances to be killed.</p>
<p>There are many more configuration settings you can pass to the Docker image, including ones to control frequency, probability, and type of terminations. Also, you can (and should) configure Chaos Monkey to send email notifications about terminations. I encourage you to <a href="https://github.com/mlafeldt/docker-simianarmy#readme">read the documentation</a> to learn more.</p>
<p>As always, it’s a good idea to <em>start small</em>. I strongly recommend testing Chaos Monkey in a staging environment before unleashing it in production.</p>
<p>This article isn’t meant to be a comprehensive guide on operating Chaos Monkey in production. However, I want to at least mention that <em>observability</em>, through monitoring and other means, is very important when it comes to chaos experiments, even more so when they’re automated. We want to eliminate customer impact as quickly as possible.</p>
<h2 id="manual-vs-automated-fault-injection">Manual vs. automated fault injection</h2>
<p>Now that running Chaos Monkey is only a single command away, should we stop manual testing altogether?</p>
<p>The answer is <em>hell, no!</em></p>
<p>Chaos Monkey is a useful tool to discover weaknesses in your systems caused by various kinds of instance failures, failures that you’d otherwise have to inject manually (or by developing your own tools).</p>
<p>GameDay events, on the other hand, bring the whole team together to think about failure modes and conduct chaos experiments, which is ideal to transfer knowledge and foster a shared mindset. It might also be the only way to test more complex scenarios that are hard or impossible to automate.</p>
<p>That being said, both manual and automated fault injection are valuable – and both have limitations. I will continue to apply these two approaches and share my experience with you.</p>
Bring your tools with you2016-04-19T00:00:00+02:00https://sharpend.io/bring-your-tools-with-you<p><img src="/assets/images/tools.jpg" alt="" /></p>
<p>Last year I joined Jimdo’s Werkzeugschmiede team as an Infrastructure Toolsmith. In brief, our goal is to provide a platform (PaaS) where all Jimdo developers can deploy and run their Dockerized applications with as little friction as possible. I’m impressed with what the team has built so far, and to be honest, also a bit surprised just how much work goes into building and running such a platform. At the same time, helping the project move forward in the right direction has been a great opportunity for me to learn and experiment with new technologies.</p>
<p>Our platform, which we internally call <em>Wonderland</em>, utilizes Amazon ECS to run Docker containers on a managed cluster of EC2 instances. For the cluster instances, we decided to use CoreOS, a container-focused Linux distribution optimized for large-scale deployments. These are the basic building blocks of our microservices infrastructure. Everything else – from providing a deployment API to integrating with external services for metrics and logging – is our job. (We also have a chat bot called Alice, and of course, it’s not the only reference to Lewis Carroll’s novel in our stack.)</p>
<p>As is usually the case with non-trivial software, there’s always something – be it ever so small – <a href="/unintended-consequences">that doesn’t work as expected</a>. The system is misbehaving in some way and you want to figure out what the heck is going on. In that situation, you instinctively fire up your favorite debugging tools, the ones you trust and are comfortable with, no matter the circumstances. The only problem: the tools you need so desperately might <em>not</em> be available in the environment you’re supposed to scrutinize. Now what?</p>
<p>You might be tempted to run <code class="language-plaintext highlighter-rouge">apt-get install</code> or <code class="language-plaintext highlighter-rouge">yum install</code> to get the programs you want, even on a production server. What could possibly go wrong? Depending on your infrastructure, the answer is either “not much” (after debugging, you throw away the cluster instance and it will be replaced automatically) or “a lot” (you keep using the now tainted system – a unique snowflake among your servers).</p>
<p>With the rise of container technology, and Docker in particular, there’s another appealing option centered around one simple idea: <strong>you bring your tools with you.</strong></p>
<h2 id="coreos-toolbox">CoreOS Toolbox</h2>
<p>As I said, something always goes wrong; our platform is no exception. For example, one time we had to debug an internal Go application that routes all Docker container output from our cluster instances to Papertrail. For some mysterious reason, the application would stop sending any logs out of the blue. (We have since moved on to using the <code class="language-plaintext highlighter-rouge">fluentd</code> logging driver, which turned out to be simpler and more reliable.)</p>
<p>In order to find out whether log data was being sent at all, I checked if <code class="language-plaintext highlighter-rouge">tcpdump</code> or <code class="language-plaintext highlighter-rouge">ngrep</code> were available under CoreOS. Unfortunately, the answer was no. Now I could have installed the missing tools inside the Docker container running the Go application (via <code class="language-plaintext highlighter-rouge">docker exec</code>). However, altering the system you want to observe is generally a bad idea as it may cause certain bugs to disappear or change their behavior (so-called Heisenbugs), making it hard to isolate the actual problem.</p>
<p>That’s one of the reasons why CoreOS prevents you from modifying the data in <code class="language-plaintext highlighter-rouge">/usr</code> by mounting it as read-only. This is where system binaries live and you’re not supposed to mess with them. Given that the root filesystem is writable, you could still install binaries to <code class="language-plaintext highlighter-rouge">/opt/bin</code>, for instance. However, doing so in an ad hoc way without using something like <code class="language-plaintext highlighter-rouge">coreos-cloudinit</code> would add another unique snowflake to your server farm. I’m afraid we’re going round in circles…</p>
<p>Of course, I wouldn’t write this article if there wasn’t some delightful solution: CoreOS comes with a helpful little script called <strong><a href="https://github.com/coreos/toolbox">toolbox</a></strong>, which will launch a container specifically for the purpose of bringing in your favorite command-line tools.</p>
<p>By default, the <code class="language-plaintext highlighter-rouge">toolbox</code> command will give you a container based on <code class="language-plaintext highlighter-rouge">fedora:latest</code>, but you may also use your own custom Docker image that comes with the tools you need. The spawned container will have full system privileges allowing you to inspect anything running on CoreOS – including other containers – with <code class="language-plaintext highlighter-rouge">ngrep</code>, <code class="language-plaintext highlighter-rouge">tcpdump</code>, and friends.</p>
<h2 id="byot">BYOT</h2>
<p>Toolbox itself might not be the most exciting engineering achievement out there. Still, it’s a good example of the <strong>bring-your-own-tools (BYOT) pattern</strong>. Now that containers are all the rage and many companies are starting to embrace new ways of running services in production, I think it makes sense to also adopt the same technology in other areas like debugging.</p>
<p>I, for my part, like the idea of having my tools at my disposal whenever I need them.</p>
Unintended Consequences2016-04-06T00:00:00+02:00https://sharpend.io/unintended-consequences<p>Here are three seemingly unrelated stories:</p>
<ul>
<li>
<p>Last week, Turkish President Erdoğan demanded the deletion of a German satirical video mocking him for his controversial actions restricting freedom of speech and other human rights in his country. The German government did not comply with his request. In fact, Erdoğan achieved the exact opposite: his reaction to the video inadvertently drew <em>further</em> public attention to it, making it all the more popular in the rest of the world. (I guess the president has never heard of <a href="https://en.wikipedia.org/wiki/Streisand_effect">Barbra Streisand</a>.)</p>
</li>
<li>
<p>Group chat is great for getting quick feedback, managing crises that require immediate attention (such as outages), and sharing funny pictures of cats. At the same time, group chat can lead to mental fatigue, fear of missing out, and an ASAP culture that is all about <em>now</em>. The companies behind popular chat tools like Slack or HipChat are unlikely to highlight these <a href="https://m.signalvnoise.com/is-group-chat-making-you-sweat/">negative consequences</a>.</p>
</li>
<li>
<p>Viagra was originally developed to lower blood pressure. Nearly any drug has side effects. Some of them are negative, others are beneficial. Viagra, the little blue pill, turned out to be the first effective treatment against erectile dysfunction – a totally unexpected outcome.</p>
</li>
</ul>
<p>What do these three stories have in common? They are examples of <strong>unintended consequences</strong>.</p>
<h2 id="not-what-you-thought">Not what you thought</h2>
<p>Unintended consequences, sometimes also called unanticipated consequences, are results that are not the ones foreseen by specific actions. In other words, you do X to achieve Y, but what you get instead, or additionally, is Z.</p>
<p><a href="https://en.wikipedia.org/wiki/Unintended_consequences">There are three types</a> of unintended consequences, and I already gave you one example for each type in the stories above:</p>
<ul>
<li><em>Unexpected benefit</em>: A positive, unexpected benefit. Viagra falls into this category. Its surprising discovery can be attributed in large parts to luck.</li>
<li><em>Unexpected drawback</em>: A negative, unexpected disadvantage occurring in addition to the desired effect. Group chat, while being a great invention, also suffers from unintended consequences such as mental exhaustion.</li>
<li><em>Perverse result</em>: A perverse effect contrary to what was originally intended. Erdoğan tried to wipe an unpleasant video from the internet – and it backfired big time.</li>
</ul>
<p>Now you might wonder how any of this fits into the overall theme of running production systems. Trust me, it matters more than you might think – but read on.</p>
<h2 id="the-challenge-of-web-systems">The challenge of web systems</h2>
<p>In web systems, or any complex software system for that matter, we have to deal with unintended consequences all the time. While we do enjoy unexpected benefits from time to time – fixing bug X accidentally solves problem Y as well – it’s the unexpected drawbacks that are more apparent, and thus more interesting. In web systems, outages are probably the most common manifestation of unintended consequences.</p>
<p><strong>The majority of outages are self-inflicted.</strong> At some point, someone did something and it had an unintended consequence. You push a bad configuration or deploy a buggy Docker image and all of a sudden the website goes down. It has happened to all of us. I, for one, have certainly caused my fair share of outages, despite being a very careful person.</p>
<p>But why is it that deliberate changes to web systems will often have negative consequences?</p>
<p>The challenge is that web systems are inherently complex. These systems are composed of many moving parts that work together to form an intricate whole. There’s a high rate of change and often a variety of processes leading to those changes. This makes it hard – if not impossible – to fully understand how all the bits and pieces resonate with each other under different conditions. Put another way, <a href="https://queue.acm.org/detail.cfm?id=2353017">web systems are largely intractable</a>, which is a major reason why outages – be they self-inflicted or not – are both unavoidable and unpredictable.</p>
<p>(Then there’s also the butterfly effect, which says that small causes can have large effects, but I won’t go into that here.)</p>
<h2 id="decouple-all-the-things">Decouple all the things</h2>
<p>So the trouble is that we’re thinking about a change we’re going to make, but we don’t necessarily anticipate the negative consequences it might have on the system as a whole. In software engineering, there’s a term for this: <em>coupling</em>. We fail to anticipate difficulties because we don’t think about how coupled a piece of software is to the rest of the system.</p>
<p>If we want to build reliable systems – systems that minimize the risk of self-inflicted outages – we have to remove the coupling. <strong>We have to decouple all the things.</strong></p>
<p>A decoupled system allows changes to be made to any one component without having an effect on any other component. By isolating each individual piece, we no longer have to keep all these complicated models in our head. Instead, we only have to know the internals of the one component we want to modify (and the interfaces it uses). This in turn reduces the probability of unintended consequences.</p>
<h2 id="example-kubernetes">Example: Kubernetes</h2>
<p>A prime example of decoupling at work is <a href="http://kubernetes.io/">Kubernetes</a>, the container cluster manager from Google.</p>
<p>Kubernetes makes it easy to build reliable distributed systems by enabling people to create <em>decoupled distributed systems</em>. Developers can, for example, talk to the Kubernetes API to deploy their application containers, and they can do so without having to worry about cluster nodes, host OS/kernel, or underlying hardware. This way, Kubernetes allows us to decouple operations and separate concerns in terms of teams.</p>
<p><strong>Kubernetes is worth a look if you want fewer outages and other surprises in your ops life.</strong></p>
<p>For more on this topic, I highly recommend watching <a href="https://www.oreilly.com/learning/how-container-clusters-like-kubernetes-change-operations">this presentation by Brendan Burns</a>. He inspired me to think about unintended consequences and write about them here.</p>
Complacency: The Enemy of Resilience2016-03-22T00:00:00+01:00https://sharpend.io/complacency-the-enemy-of-resilience<p><img src="/assets/images/complacency.jpg" alt="" /></p>
<p>Being on-call can be a nerve-wracking experience. Something might break any minute. It’s unpredictable. Yet, you’re in charge when a server explodes (and they like to explode in the middle of the night). Knowing this might freak you out. At least that’s what happened to me when I was on-call for the first couple of times after joining Jimdo. As a bloody beginner, I was afraid of what <em>might</em> happen. I was afraid of that buzzing alert sound to ever go off. Would I be able to handle the situation with the little training I had? All I wanted was to survive whatever on-call shift I was on. It stressed me out.</p>
<p>Before moving to the cloud, every engineer who participated in our on-call rotation had to deal with a relatively large number of incidents over time – most of them minor in severity but high on the annoyance scale (<em>lighttpd backend overloaded</em> anyone?). Due to the nature of our architecture, we often experienced cascading failures – failures that spread from one part of the system to another, leading to a good deal of alerts for us to handle. Once a server was in production, we tried to touch it as little as possible – unless something was broken, of course. And things did break all the time.</p>
<p>That was three years ago. A lot has changed since then. We finished migrating our core infrastructure from bare metal to the cloud. AWS is doing an amazing job when it comes to reliability. This and the fact that we had to redesign our software for the cloud – and yes, we no longer use lighttpd – led us to where we are today: outages are rare, our customers are happier, and we get a lot more sleep.</p>
<p>So all is good now, right?</p>
<h2 id="when-failure-is-the-rule">When failure is the rule</h2>
<p>It is beyond question that having too many alerts – and fighting too many fires at the same time – is a real problem, as it might lead to <a href="https://en.wikipedia.org/wiki/Alarm_fatigue">alert fatigue</a>, sleep deprivation, and a good amount of frustration. In other words, plenty of stressful situations that used to keep us busy and distracted us from our actual work.</p>
<p>However, the fact that we were dealing with failure on a regular basis also had some advantages:</p>
<ul>
<li>Everyone knew, for the most part, what to do when shit hit the fan. We all had experience with the most common failure modes. Like a well-oiled machine, we followed pretty much the same procedures when it came to incident communication, documentation, escalation, etc.</li>
<li>Due to the sheer amount of alerts, we knew that our monitoring and alerting systems worked in principle (leaving aside the fact that some alerts were questionable). These days, alerts are so rare that I sometimes wonder if PagerDuty is broken.</li>
<li>Last, but most importantly, we never believed that our system was particularly resilient to failures. On the contrary, we were always wary of what might happen next.</li>
</ul>
<h2 id="when-failure-is-the-exception">When failure is the exception</h2>
<p>On the other end of the spectrum, when failure is the exception rather than the rule, you risk losing every single advantage I outlined above. As John Allspaw put it in his article, <a href="https://queue.acm.org/detail.cfm?id=2353017">Fault Injection in Production</a>:</p>
<blockquote>
<p>If a system has a period of little or no degradation, there is a real risk of it drifting toward failure on multiple levels, because engineers can become convinced – falsely – that the system is experiencing no problems because it is inherently safe.</p>
</blockquote>
<p>Just because there are no problems now doesn’t mean it’s going to stay that way. Sooner or later, any complex system will fail. It’s inevitable. That’s why you should never get too comfortable.</p>
<p>There’s a word for this very condition: <em>complacency</em>. Merriam-Webster <a href="http://www.merriam-webster.com/dictionary/complacency">defines complacency</a> as “self-satisfaction especially when accompanied by <strong>unawareness of actual dangers or deficiencies</strong>”.</p>
<p>Complacency is the enemy of resilience. The longer you wait for disaster to strike in production – merely hoping that everything will be okay – the less likely you are to handle emergencies well, both at a technical and organizational level.</p>
<h2 id="how-to-fight-complacency">How to fight complacency</h2>
<p>Ultimately, our goal is to be aware of actual dangers and to prepare for them accordingly. Learning from outages after the fact is important (and that’s what postmortems are for), but it shouldn’t be the only method for acquiring operational knowledge. Allspaw agrees, as he continues:</p>
<blockquote>
<p>[…] building resilient systems requires experience with failure, and that we want to anticipate and confirm our expectations surrounding failure <em>more</em> often, not <em>less</em> often. Shying away from the effects of failure in a misguided attempt to reduce risk will result in poor designs, stale recovery skills, and a false sense of safety.</p>
</blockquote>
<p>This observation may come as a surprise for those who believe that systems should never fail, but waiting for things to break in production is <em>not</em> an option. We should rather inject failures proactively in a controlled manner in order to prepare for the worst and gain confidence in our systems. This is the core idea behind <a href="/chaos-engineering-101">Chaos Engineering</a> and related practices like GameDay exercises. Again, Allspaw summed it up best:</p>
<blockquote>
<p>[…] failure-inducing exercises can serve as “vaccines” to improve the safety of a system—a small amount of failure injected to help the system learn to recover. It also keeps a concern about failure alive in the culture of engineering teams, and it keeps complacency at bay.</p>
</blockquote>
<p>These days, we’re running GameDay exercises periodically at Jimdo – making sure we never get too comfortable. We know that, once again, <a href="/chaos-engineering-a-shift-in-mindset">embracing failure</a> is key to building resilient systems.</p>
How to succeed at infrastructure automation2016-03-07T00:00:00+01:00https://sharpend.io/how-to-succeed-at-infrastructure-automation<p>I care a lot about infrastructure automation and the art of turning infrastructure into code. I find pleasure in using <em>and</em> developing build, test, and deployment systems. It’s what I get paid to do every day and <a href="/infrastructure-automation-by-example">can’t stop doing after work</a>. In a sense, I’m obsessed with automation.</p>
<p>For an engineer like me, the questions of <em>what</em> to automate and <em>how</em> to go about it are of particular interest. Alas, I don’t always have the right answer to those questions. That’s when things go wrong and mistakes happen.</p>
<p>Fortunately, mistakes are also a wonderful opportunity to learn – and to eventually succeed.</p>
<h2 id="screwing-up">Screwing up</h2>
<p>I’ve been automating tasks of one kind or another for about a decade now. It goes without saying that I made some rookie mistakes along the way. I don’t mean blunders like pushing buggy code or executing commands in the wrong environment (oops). These things are unavoidable. What I mean are more fundamental problems that go beyond mere technical matters.</p>
<p>Let me share three personal anecdotes about screwing up in one way or another:</p>
<p>I once spent three full days writing automated tests for a shell script that performs backups. I wanted to change the script – a beast of hundreds of lines of messy code – without breaking the backup process. After wasting a couple hours trying to tame the test framework to do what I wanted it to do (how hard could it be?), I already knew that this wasn’t going to end well. While I eventually managed to write the tests, they ended up being very brittle and verbose, adding only little confidence. Even worse – and hindsight bias aside – at no time did those three days feel like adding any value. It was just my ego pushing me.</p>
<p>The first configuration management system <a href="/blog/learning-chef">I learned to use</a> – and still the one I like the most – is Chef. I have a thing for Chef because it’s based on Ruby, which happens to be one of my favorite programming languages. When I started working at Jimdo in 2013, however, I suddenly had to use Puppet. Instead of coming to grips with Puppet and accepting the situation as it was, I spent my first weeks at the new gig ranting about how bad this Puppet thing is and how Chef would magically solve all problems. Of course, this didn’t change anything. I knew that sooner or later I had to learn Puppet if I wanted to work on Jimdo’s infrastructure.</p>
<p>The last mistake is the worst of the three. <em>I failed to deliver</em> because I didn’t care enough about a project as a whole. The project’s goal was to add proper monitoring to our cron jobs. For this, we developed <a href="https://github.com/Jimdo/periodicnoise">a tool in Go</a> that would allow us to wrap cron jobs and send results to Nagios. It worked out pretty well. The problem: building that tool was only a small but by far the most interesting part of the project. Rather than wrapping up the remaining tasks, I was too busy learning more about the shiny technologies we explored at that time (Go and AWS). This isn’t the full story; suffice it to say I wasn’t part of the team that completed the project later on.</p>
<h2 id="the-three-rules-of-infrastructure-automation">The three rules of infrastructure automation</h2>
<p>It goes without saying that I don’t want to repeat any of those mistakes. To achieve that, I’ve turned the lessons I learned over the years into three simple rules. I call them, appropriately, <em>The three rules of infrastructure automation</em>. I’m convinced that by following these rules, I’ve been able to improve my work and become a more successful engineer.</p>
<p>The three rules are:</p>
<ol>
<li>
<p><strong>Don’t blindly automate all the things.</strong> Take a step back and evaluate if a task is actually worth the effort. <em>Feel the pain</em> before doing something about it. Ignore the problem for a while; it might not be that big of an issue. If you still decide to invest in automation, you will know exactly where the pain points are. Ask for feedback when you’re stuck. Always be willing to adjust and, more importantly, have the courage to stop what you’re doing and move on.</p>
</li>
<li>
<p><strong>Treat tools as what they are: a means to an end.</strong> Don’t fight over tools or programming languages. Use whatever works for you or your company. In the end, it’s all about creating (business) value. More often than not, tools are not the reason why we fail to reach our goals. Before complaining about software, remember that certain design decisions probably made sense to the people at the time they built it. Be open to learning new things.</p>
</li>
<li>
<p><strong>Take ownership of your work.</strong> First and foremost, do the work and deliver what you promise (while adhering to the first two rules). Get both the interesting <em>and</em> the boring tasks done. Yes, automation isn’t always fun. That’s just how it is. Accept it and move forward. Taking ownership also means to keep caring. Fix bugs. Help others relying on your work. Don’t automate and forget.</p>
</li>
</ol>
<p>I’ve learned to live my professional life by the above rules. They provide me with the right mental framework for approaching automation, if not programming in general. These rules were and continue to be a tremendous help to me. They may help you too.</p>
Chaos Engineering: A Shift in Mindset2016-02-23T00:00:00+01:00https://sharpend.io/chaos-engineering-a-shift-in-mindset<p><img src="/assets/images/chaos-engineering-mindset.jpg" alt="" /></p>
<p>Last time <a href="/chaos-engineering-101">I introduced you to Chaos Engineering</a>, a discipline based on the idea that proactively triggering failures in a system is the best way to prepare for disaster.</p>
<p>My goal was to show you how to get started running your own chaos experiments, mostly on a technical level. I left out some things by necessity. For example, I only briefly mentioned that it would be a good idea to introduce a company to the concept of Chaos Engineering by starting small, rather than wreaking havoc on production from the get-go. I took it as given that chaos experiments are endorsed by everyone. Of course, that’s not the case.</p>
<p>Chaos Engineering, or engineering in general, doesn’t happen in a vacuum. There are people involved. Chaos Engineering isn’t merely a technical solution to the problem of building resilient systems. For it to work at all, it requires a fundamental <em>shift in the mindset</em> of managers and employees, if not whole companies.</p>
<h2 id="failure-is-a-hard-sell">Failure is a hard sell</h2>
<p>For many companies, failure is unacceptable. Management considers outages, big or small, to be intolerable. They believe that systems should never fail. And they spend a lot of money in trying to make this dream a reality (while never getting there). Most of their managers, especially those of non-technical departments, will argue that chaos experiments offer little benefit and carry a substantial risk.</p>
<p>In these companies, people are more likely to cover up a mistake than deal with it openly because they’re afraid to be punished for failure. This blame culture is seriously toxic. Not only does it hurt morale, it also hinders innovation through collaboration, thus depriving the opportunity to learn from mistakes together.</p>
<p>And it’s not just managers. For engineers who try to avoid emergencies altogether, it too is difficult to stand by as their systems break in ways they can’t possibly imagine. After all, a deployment pipeline running automated tests ought to be enough, right?</p>
<p>Introducing Chaos Engineering – forcing systems to fail – seems to be impossible in such circumstances. Indeed, it would take more than a technological shift. It would need <em>a change in culture.</em></p>
<h2 id="learning-to-embrace-failure">Learning to embrace failure</h2>
<p>Luckily, there are other companies, like Etsy or Stripe, that have learned to embrace failure. They realize that failure is unavoidable, whether we’re developing software or managing people or doing something else entirely. They understand that the key to building resilient systems is to accept that failure is a part of life. Unsurprisingly, these companies are the first to adopt Chaos Engineering or similar practices, and therefore are better equipped to manage outages and other surprises once they <em>do</em> happen.</p>
<p>I know this because I’m privileged to be working in <a href="https://www.jimdo.com/">a place</a> where failure isn’t a disaster, but a learning experience. I wouldn’t go so far as to say that failure is part of our company culture – not yet – but engineers are trusted to do the right thing, with plenty of room for (chaos) experiments. Besides trying to anticipate where something might break in the future, we’re also having blameless postmortems on outages and accidents as a means to learn from past events.</p>
<p>In fact, it happened more than once that I was on-call when something bad happened which I eventually had to escalate to my boss. In the end, we always fixed problems by working together as a team. The inevitable postmortems that followed helped us understand how accidents actually happened – without blaming individuals – and how we might better prepare for the future.</p>
<h2 id="how-to-make-the-case-for-chaos-experiments">How to make the case for chaos experiments</h2>
<p>Again, I know I’m in a privileged position. You might not be so lucky. The challenge is to convince people – the <em>right</em> people – of the value of Chaos Engineering. You want them to see how much there is to learn from failure and that the benefits, such as increased confidence in the system, outweigh the costs.</p>
<p>If you’re having a hard time selling the idea of Chaos Engineering, here are a couple things that may or may not work for you:</p>
<ul>
<li>Look out for outages and other failures. They can give you the ammunition you need to get the approval for some smaller chaos experiments.</li>
<li>Try to find a sponsor, someone in power who understands the value of resilience testing and is willing to promote it throughout the company.</li>
<li>Suggest to experiment in a non-production environment first (as mentioned before). Even though there are downsides to this – some behaviors can be seen only in production; repairing a broken test environment is often a lot of work – you can expect to get less pushback.</li>
<li>Make your systems fault-tolerant to the best of your ability <em>before</em> running chaos experiments. I hope this is obvious.</li>
<li>Mitigate risk as much as possible and communicate your efforts. Make sure to carefully review experiments, monitor them closely, and have experts at hand who can revert changes should things go wrong.</li>
<li>Keep stakeholders in the loop. Announce experiments early enough. They can be risky and disruptive. You don’t want to lose trust before having a real chance to earn it.</li>
<li>Share insights with other teams. Help them get started with their own experiments. More teams are likely to join once they see the results.</li>
<li>If successful, <em>tell everyone</em> how Chaos Engineering was instrumental in improving resilience. Advertise it as much as you can!</li>
</ul>
<p>I hope this helps.</p>
<p>Building resilient systems is an ongoing, iterative process. Making a compelling case for Chaos Engineering is a good first step in changing the prevailing mindset that failure is a bad thing.</p>
<h2 id="further-reading">Further reading</h2>
<p>This piece was, to a great extent, inspired by these ACM articles:</p>
<ul>
<li><a href="https://queue.acm.org/detail.cfm?id=2371297">Resilience Engineering: Learning to Embrace Failure</a> – a discussion with Jesse Robbins, Kripa Krishnan, John Allspaw, and Tom Limoncelli</li>
<li><a href="https://queue.acm.org/detail.cfm?id=2499552">The Antifragile Organization</a> by Ariel Tseitlin</li>
<li><a href="https://queue.acm.org/detail.cfm?id=2371516">Weathering the Unexpected</a> by Kripa Krishnan</li>
<li><a href="https://queue.acm.org/detail.cfm?id=2353017">Fault Injection in Production</a> by John Allspaw</li>
</ul>
<p>I strongly recommend reading all of them if you want to dig deeper.</p>
Chaos Engineering 1012016-02-10T00:00:00+01:00https://sharpend.io/chaos-engineering-101<p><img src="/assets/images/chaos-engineering-101.jpg" alt="" /></p>
<p>Say you’re developing a new web application – the next great thing everybody has been waiting for. After all the work you’ve done, it’s time to finally launch the service to the first customers. Now the hard question:</p>
<p><strong>When do you know that the application is ready for production? More specifically, how can you be sure that the (distributed) system you’ve built is resilient enough to survive use in production?</strong></p>
<p>The truth is: you can never be sure. You don’t know what’s going to happen. There will always be something that can – <a href="https://github.com/danluu/post-mortems">and will</a> – go wrong, from self-inflicted outages caused by bad configuration pushes or buggy images to events that are outside your control like denial-of-service attacks or network failures. No matter how hard you try, you can’t build perfect software (or hardware, for that matter). Nor can the companies you depend on.</p>
<p>We live in an imperfect world. That’s just how it is. Accept it and focus on the things you can control: creating a quality product that is <em>resilient to failures</em>. Build software that is able to cope with both expected and unexpected events; gracefully degrade whenever necessary. As the saying goes, “Hope for the best and prepare for the worst”.</p>
<p>But how? How can you make sure you’re ready for disaster?</p>
<p>The first thing you need to do is to identify problems that could arise in production. Only then will you be able to address systemic weaknesses and make your systems fault-tolerant.</p>
<p>This is where <strong>Chaos Engineering</strong> comes in.</p>
<h2 id="principles-of-chaos-engineering">Principles of Chaos Engineering</h2>
<p>Rather than waiting for things to break in production at the worst time, the core idea of Chaos Engineering is to proactively inject failures in order to be prepared when disaster strikes.</p>
<p>Netflix, a pioneer in the field of automated failure testing (and, by the way, also a great video-streaming service), <a href="https://principlesofchaos.org/">defines Chaos Engineering</a> as follows:</p>
<blockquote>
<p>Chaos Engineering is the discipline of experimenting on a distributed system in order to build confidence in the system’s capability to withstand turbulent conditions in production.</p>
</blockquote>
<p>As a Chaos Engineer, you test a system’s ability to survive failures by simulating potential errors (aka failure modes) in a series of controlled experiments. These experiments typically consist of four steps:</p>
<ol>
<li>Define the system’s normal behavior – its “steady state” – based on measurable output like overall throughput, error rates, latency, etc.</li>
<li>Hypothesize about the steady state behavior of an experimental group, as compared to a stable control group.</li>
<li>Expose the experimental group to simulated real-world events such as server crashes, malformed responses, or traffic spikes.</li>
<li>Test the hypothesis by comparing the steady state of the control group and the experimental group. The smaller the differences, the more confidence we have that the system is resilient.</li>
</ol>
<p>Or, to put it in less scientific terms: <strong>intentionally break things, compare measured with expected impact, and correct any problems uncovered this way.</strong></p>
<p>As an example, let’s say you want to know what happens if, for some reason, your MySQL database isn’t available. You hypothesize that, in this case, your web application would stop serving requests, immediately returning an error instead. To simulate the event, you block access to the database server. Afterwards, however, you observe that the app seems to take forever to respond. After some investigation, you find the cause – a misconfigured timeout – and fix it in a matter of minutes.</p>
<p>As this example demonstrates, Chaos Engineering makes for effective resilience testing. Besides, it’s a ton of fun, but read on.</p>
<h2 id="how-to-get-started">How to get started</h2>
<p>Netflix went the extra mile and built several autonomous agents, so-called “monkeys”, for injecting failures and creating different kinds of outages. For example, Chaos Monkey randomly terminates virtual machines, Latency Monkey induces artificial delays in API calls to simulate service degradation, and Chaos Gorilla is programmed to take down an entire datacenter. Together they form the <a href="https://queue.acm.org/detail.cfm?id=2499552">Simian Army.</a></p>
<p>While the Simian Army might be a novel concept, you don’t need to automate experiments to run continuously when you’re just getting started. Besides, it is best to introduce a company to the concept of Chaos Engineering by starting small.</p>
<p>So rather than wreaking havoc on your production system from day one, start by experimenting in an isolated staging environment (if you don’t have a pre-production environment yet, now would be the perfect time to create one). While the two environments are likely to be different in more or less subtle ways, any resilience testing is better than no resilience testing. Later, when you feel more confident, conduct some of the experiments – or preferably all of them – in production. Remember: Chaos Engineering is focused on <em>controlled failure-injection.</em> You make the rules!</p>
<p>The purpose of our chaos experiments is to simulate disaster conditions. This might sound like a difficult task – and yes, it does require a lot of creativity – but in the beginning it’s easiest to focus on <em>availability</em>, or rather the lack of it: inject failures so that certain pieces of your infrastructure become unavailable. Intentionally terminate cluster machines, kill worker processes, delete database tables, cut off access to internal and external services, etc. This way you can learn a lot about the coupling of your system and discover subtle dependencies you would otherwise overlook.</p>
<p>Later on you might want to simulate other events capable of disrupting steady state, like high latency caused by slow network performance. These experiments are generally harder to pull off and often require special tooling, but the takeaways are worth the extra effort.</p>
<p>Whatever you decide to do, you’ll be surprised how much you can learn from chaos.</p>
<h2 id="plan-execute-measure-adjust">Plan, execute, measure, adjust</h2>
<p>Briefly, here are the steps involved in conducting chaos experiments. This list is based on my own experience in participating in <a href="https://queue.acm.org/detail.cfm?id=2371297">GameDay events</a> at Jimdo.</p>
<ol>
<li>Start by planning the experiments. Compile a list of potential failure modes, how you want to simulate them, and what you think the impact will be. I recommend using a spreadsheet.</li>
<li>Pick a date. Inform stakeholders of affected systems, especially if you anticipate any trouble for customers.</li>
<li>Gather the whole team in front of a big screen and go through the experiments together. This is the best way to transfer knowledge and develop a shared mindset.</li>
<li>After each experiment, write down the actual measured impact.</li>
<li>For each discovered flaw, put together a list of counter measures. Don’t implement them right away! Add any follow-up items to your issue tracker.</li>
</ol>
<p>Make sure to repeat the experiments on a regular basis (at least once every quarter) to detect regressions as well as new problems. Don’t forget to bring your spreadsheet.</p>
<hr />
<p>I hope you enjoyed this introduction to Chaos Engineering – a powerful, if somewhat radical, approach to building resilient systems.</p>
<p>As always, the best way to internalize new concepts is by practice. Therefore, start running your own chaos experiments! It’s well worth it.</p>
<p><em>Update: Since writing this article, my understanding of Chaos Engineering has evolved. Read <a href="/the-discipline-of-chaos-engineering">The Discipline of Chaos Engineering</a> for a more recent take on the topic.</em></p>
Embedding Assets in Go (Codeship)2015-11-05T00:00:00+01:00https://sharpend.io/embedding-assets-in-go<p><img src="/assets/images/gopher-biplane.jpg" alt="Image by Renee French" /></p>
<p>I wrote another article for Codeship: <strong><a href="https://blog.codeship.com/embedding-assets-in-go/">Embedding assets in Go</a></strong></p>
<p>If the story sounds familiar, it’s because the article is based on a <a href="/blog/embedding-assets-in-go">blog post</a> of mine with the same title. I added lots of content, almost doubling the number of words in the process. I also tried to improve the overall reading experience by fixing what I thought would be good writing back when I conceived the original piece. (Turns out revising old posts is <em>very</em> similar to refactoring legacy code in the amount of pain you will experience.)</p>
<p>I hope you will enjoy my latest Codeship article.</p>
<p>Have a different or better approach? Reach out to me <a href="https://twitter.com/mlafeldt">on Twitter</a>.</p>
Using Docker to Build Debian Packages (Codeship)2015-08-18T00:00:00+02:00https://sharpend.io/using-docker-to-build-debian-packages<p>TL;DR: <a href="https://blog.codeship.com/using-docker-build-debian-packages/">I wrote an article</a> for the Codeship blog.</p>
<p>About two months ago, <a href="https://twitter.com/manualwise">Manuel Weiss</a> asked me if I would be interested in writing original content for the <a href="http://blog.codeship.com/">Codeship blog</a>. Manuel is one of Codeship’s co-founders; I felt honored that he reached out to me. I was thrilled by the opportunity to guest post on a well-known tech blog, which is something that aligns well with <a href="/blog/writing-goals/">my writing goals</a>. (To be honest, getting paid for my work certainly didn’t hurt either.)</p>
<p>Now all I needed was an engaging topic for the blog’s audience. <a href="https://www.docker.com/">Docker</a> is all the rage these days and I already had <a href="/blog/rethinking-package-building-at-jimdo/">a lot of experience</a> in using it for building Debian packages. And so, unsurprisingly, I decided to write about that. I crafted a short outline and pitched it to their editor. They accepted my idea and I was off writing…</p>
<p>Head over to the Codeship blog and read the final article, <strong><a href="https://blog.codeship.com/using-docker-build-debian-packages/">Using Docker to build Debian packages</a></strong>, to learn how and why I developed a build system for Debian packages at Jimdo.</p>
<p>PS: While I finished the article in time, I made my life much harder than it had to be. Let me give you this piece of advice: working on an article with a deadline and moving to another home should <em>not</em> be done at the same time.</p>
<h2 id="update-reactions">Update: Reactions</h2>
<p>Looks like my first article for Codeship has been well received. It was featured in <a href="http://goto.docker.com/Docker-Weekly-08192015.html">Docker Weekly</a> and <a href="http://email.changelog.com/t/t-E3FCCD1BFEFBDDCD">Changelog Weekly</a>. People also said some nice things about it on Twitter:</p>
<blockquote class="twitter-tweet" data-cards="hidden" lang="en"><p lang="en" dir="ltr">"Using Docker to Build Debian Packages" - via <a href="https://twitter.com/mlafeldt">@mlafeldt</a>. Thank you, Mathias! <a href="http://t.co/UYJsyH1GF7">http://t.co/UYJsyH1GF7</a> <a href="http://t.co/MkOgj3OoEj">pic.twitter.com/MkOgj3OoEj</a></p>— Codeship (@codeship) <a href="https://twitter.com/codeship/status/633666084105768961">August 18, 2015</a></blockquote>
<blockquote class="twitter-tweet" data-cards="hidden" lang="en"><p lang="en" dir="ltr">Proud to have <a href="https://twitter.com/mlafeldt">@mlafeldt</a> publish a post for our blog. I love his writing! "Using <a href="https://twitter.com/docker">@Docker</a> to Build Debian Packages" <a href="http://t.co/leo2g45q3h">http://t.co/leo2g45q3h</a></p>— Manuel Weiss (@manualwise) <a href="https://twitter.com/manualwise/status/633686998235107329">August 18, 2015</a></blockquote>
<blockquote class="twitter-tweet" data-cards="hidden" lang="en"><p lang="en" dir="ltr">Awesome post on building <a href="https://twitter.com/debian">@Debian</a> packages w/ <a href="https://twitter.com/docker">@Docker</a> by <a href="https://twitter.com/mlafeldt">@mlafeldt</a> via <a href="https://twitter.com/codeship">@codeship</a>: <a href="http://t.co/KlzWGX77gM">http://t.co/KlzWGX77gM</a> <a href="https://twitter.com/hashtag/DockerWeekly?src=hash">#DockerWeekly</a> <a href="http://t.co/mhXrVV6ZFI">pic.twitter.com/mhXrVV6ZFI</a></p>— Docker (@docker) <a href="https://twitter.com/docker/status/635094224422203392">August 22, 2015</a></blockquote>
Cook your own packages: Getting more out of fpm (SysAdvent)2014-12-15T00:00:00+01:00https://sharpend.io/cook-your-own-packages<p>When it comes to building packages, there is one particular tool that has grown in popularity over the last years: <a href="https://github.com/jordansissel/fpm">fpm</a>. fpm’s honorable goal is to make it as simple as possible to create native packages for multiple platforms, all without having to learn the intricacies of each distribution’s packaging format (.deb, .rpm, etc.) and tooling.</p>
<p>With a single command, fpm can build packages from a variety of sources including Ruby gems, Python modules, tarballs, and plain directories. Here’s a quick example showing you how to use the tool to create a Debian package of the <a href="https://github.com/aws/aws-sdk-ruby">AWS SDK for Ruby</a>:</p>
<figure class="highlight"><pre><code class="language-sh" data-lang="sh"><span class="nv">$ </span>fpm <span class="nt">-s</span> gem <span class="nt">-t</span> deb aws-sdk
Created package <span class="o">{</span>:path<span class="o">=></span><span class="s2">"rubygem-aws-sdk_1.59.0_all.deb"</span><span class="o">}</span></code></pre></figure>
<p>It is this simplicity that makes fpm so popular. Developers are able to easily distribute their software via platform-native packages. Businesses can manage their infrastructure on their own terms, independent of upstream vendors and their policies. All of this has been possible before, but never with this little effort.</p>
<p>In practice, however, things are often more complicated than the one-liner shown above. While it is absolutely possible to provision production systems with packages created by fpm, it will take some work to get there. The tool can only help you so far.</p>
<p>In this post we’ll take a look at several best practices covering: dependency resolution, reproducible builds, and infrastructure as code. All examples will be specific to Debian and Ruby, but the same lessons apply to other platforms/languages as well.</p>
<h2 id="resolving-dependencies">Resolving dependencies</h2>
<p>Let’s get back to the AWS SDK package from the introduction. With a single command, fpm converts the <code class="language-plaintext highlighter-rouge">aws-sdk</code> Ruby gem to a Debian package named <code class="language-plaintext highlighter-rouge">rubygem-aws-sdk</code>. This is what happens when we actually try to install the package on a Debian system:</p>
<figure class="highlight"><pre><code class="language-sh" data-lang="sh"><span class="nv">$ </span><span class="nb">sudo </span>dpkg <span class="nt">--install</span> rubygem-aws-sdk_1.59.0_all.deb
...
dpkg: dependency problems prevent configuration of rubygem-aws-sdk:
rubygem-aws-sdk depends on rubygem-aws-sdk-v1 <span class="o">(=</span> 1.59.0<span class="o">)</span><span class="p">;</span> however:
Package rubygem-aws-sdk-v1 is not installed.
...</code></pre></figure>
<p>As we can see, our package can’t be installed due to a missing dependency (<code class="language-plaintext highlighter-rouge">rubygem-aws-sdk-v1</code>). Let’s take a closer look at the generated .deb file:</p>
<figure class="highlight"><pre><code class="language-sh" data-lang="sh"><span class="nv">$ </span>dpkg <span class="nt">--info</span> rubygem-aws-sdk_1.59.0_all.deb
...
Package: rubygem-aws-sdk
Version: 1.59.0
License: Apache 2.0
Vendor: Amazon Web Services
Architecture: all
Maintainer: <vagrant@wheezy-buildbox>
Installed-Size: 5
Depends: rubygem-aws-sdk-v1 <span class="o">(=</span> 1.59.0<span class="o">)</span>
Provides: rubygem-aws-sdk
Section: Languages/Development/Ruby
Priority: extra
Homepage: http://aws.amazon.com/sdkforruby
Description: Version 1 of the AWS SDK <span class="k">for </span>Ruby. Available as both <span class="sb">`</span>aws-sdk<span class="sb">`</span> and <span class="sb">`</span>aws-sdk-v1<span class="sb">`</span><span class="nb">.</span>
Use <span class="sb">`</span>aws-sdk-v1<span class="sb">`</span> <span class="k">if </span>you want to load v1 and v2 of the Ruby SDK <span class="k">in </span>the same
application.</code></pre></figure>
<p>fpm did a great job at populating metadata fields such as package name, version, license, and description. It also made sure that the <code class="language-plaintext highlighter-rouge">Depends</code> field contains all required dependencies that have to be installed for our package to work properly. Here, there’s only one direct dependency – the one we’re missing.</p>
<p>While fpm goes to great lengths to provide proper dependency information – and this is not limited to Ruby gems – it does <em>not</em> automatically build those dependencies. That’s our job. We need to find a set of compatible dependencies and then tell fpm to build them for us.</p>
<p>Let’s build the missing <code class="language-plaintext highlighter-rouge">rubygem-aws-sdk-v1</code> package with the exact version required and then observe the next dependency in the chain:</p>
<figure class="highlight"><pre><code class="language-sh" data-lang="sh"><span class="nv">$ </span>fpm <span class="nt">-s</span> gem <span class="nt">-t</span> deb <span class="nt">-v</span> 1.59.0 aws-sdk-v1
Created package <span class="o">{</span>:path<span class="o">=></span><span class="s2">"rubygem-aws-sdk-v1_1.59.0_all.deb"</span><span class="o">}</span>
<span class="nv">$ </span>dpkg <span class="nt">--info</span> rubygem-aws-sdk-v1_1.59.0_all.deb | <span class="nb">grep </span>Depends
Depends: rubygem-nokogiri <span class="o">(>=</span> 1.4.4<span class="o">)</span>, rubygem-json <span class="o">(>=</span> 1.4<span class="o">)</span>, rubygem-json <span class="o">(<<</span> <span class="no">2</span><span class="sh">.0)</span></code></pre></figure>
<p>Two more packages to take care of: <code class="language-plaintext highlighter-rouge">rubygem-nokogiri</code> and <code class="language-plaintext highlighter-rouge">rubygem-json</code>. By now, it should be clear that resolving package dependencies like this is no fun. There must be a better way.</p>
<p>In the Ruby world, <a href="http://bundler.io/">Bundler</a> is the tool of choice for managing and resolving gem dependencies. So let’s ask Bundler for the dependencies we need. For this, we create a <code class="language-plaintext highlighter-rouge">Gemfile</code> with the following content:</p>
<figure class="highlight"><pre><code class="language-ruby" data-lang="ruby"><span class="c1"># Gemfile</span>
<span class="n">source</span> <span class="s2">"https://rubygems.org"</span>
<span class="n">gem</span> <span class="s2">"aws-sdk"</span><span class="p">,</span> <span class="s2">"= 1.59.0"</span>
<span class="n">gem</span> <span class="s2">"nokogiri"</span><span class="p">,</span> <span class="s2">"~> 1.5.0"</span> <span class="c1"># use older version of Nokogiri</span></code></pre></figure>
<p>We then instruct Bundler to resolve all dependencies and store the resulting .gem files into a local folder:</p>
<figure class="highlight"><pre><code class="language-sh" data-lang="sh"><span class="nv">$ </span>bundle package
...
Updating files <span class="k">in </span>vendor/cache
<span class="k">*</span> json-1.8.1.gem
<span class="k">*</span> nokogiri-1.5.11.gem
<span class="k">*</span> aws-sdk-v1-1.59.0.gem
<span class="k">*</span> aws-sdk-1.59.0.gem</code></pre></figure>
<p>We specifically asked Bundler to create .gem files because fpm can convert them into Debian packages in a matter of seconds:</p>
<figure class="highlight"><pre><code class="language-sh" data-lang="sh"><span class="nv">$ </span>find vendor/cache <span class="nt">-name</span> <span class="s1">'*.gem'</span> | xargs <span class="nt">-n1</span> fpm <span class="nt">-s</span> gem <span class="nt">-t</span> deb
Created package <span class="o">{</span>:path<span class="o">=></span><span class="s2">"rubygem-aws-sdk-v1_1.59.0_all.deb"</span><span class="o">}</span>
Created package <span class="o">{</span>:path<span class="o">=></span><span class="s2">"rubygem-aws-sdk_1.59.0_all.deb"</span><span class="o">}</span>
Created package <span class="o">{</span>:path<span class="o">=></span><span class="s2">"rubygem-json_1.8.1_amd64.deb"</span><span class="o">}</span>
Created package <span class="o">{</span>:path<span class="o">=></span><span class="s2">"rubygem-nokogiri_1.5.11_amd64.deb"</span><span class="o">}</span></code></pre></figure>
<p>As a final test, let’s install those packages…</p>
<figure class="highlight"><pre><code class="language-sh" data-lang="sh"><span class="nv">$ </span><span class="nb">sudo </span>dpkg <span class="nt">-i</span> <span class="k">*</span>.deb
...
Setting up rubygem-json <span class="o">(</span>1.8.1<span class="o">)</span> ...
Setting up rubygem-nokogiri <span class="o">(</span>1.5.11<span class="o">)</span> ...
Setting up rubygem-aws-sdk-v1 <span class="o">(</span>1.59.0<span class="o">)</span> ...
Setting up rubygem-aws-sdk <span class="o">(</span>1.59.0<span class="o">)</span> ...</code></pre></figure>
<p>…and verify that the AWS SDK actually can be used by Ruby:</p>
<figure class="highlight"><pre><code class="language-sh" data-lang="sh"><span class="nv">$ </span>ruby <span class="nt">-e</span> <span class="s2">"require 'aws-sdk'; puts AWS::VERSION"</span>
1.59.0</code></pre></figure>
<p>Win!</p>
<p>The purpose of this little exercise was to demonstrate one effective approach to resolving package dependencies for fpm. By using Bundler – the best tool for the job – we get fine control over all dependencies, including transitive ones (like Nokogiri, see <code class="language-plaintext highlighter-rouge">Gemfile</code>). Other languages provide similar dependency tools. We should make use of language specific tools whenever we can.</p>
<h2 id="build-infrastructure">Build infrastructure</h2>
<p>After learning how to build all packages that make up a piece of software, let’s consider how to integrate fpm into our build infrastructure. These days, with the rise of the DevOps movement, many teams have started to manage their own infrastructure. Even though each team is likely to have unique requirements, it still makes sense to share a company-wide build infrastructure, as opposed to reinventing the wheel each time someone wants to automate packaging.</p>
<p>Packaging is often only a small step in a longer series of build steps. In many cases, we first have to build the software itself. While fpm supports multiple source formats, it doesn’t know how to build the source code or determine dependencies required by the package. Again, that’s our job.</p>
<p>Creating a consistent build and release process for different projects across multiple teams is hard. Fortunately, there’s another tool that does most of the work for us: <a href="https://github.com/bernd/fpm-cookery">fpm-cookery</a>. fpm-cookery sits on top of fpm and provides the missing pieces to create a reusable build infrastructure. Inspired by projects like <a href="http://brew.sh/">Homebrew</a>, fpm-cookery builds packages based on simple recipes written in Ruby.</p>
<p>Let’s turn our attention back to the AWS SDK. Remember how we initially converted the gem to a Debian package? As a warm up, let’s do the same with fpm-cookery. First, we have to create a <code class="language-plaintext highlighter-rouge">recipe.rb</code> file:</p>
<figure class="highlight"><pre><code class="language-ruby" data-lang="ruby"><span class="c1"># recipe.rb</span>
<span class="k">class</span> <span class="nc">AwsSdkGem</span> <span class="o"><</span> <span class="no">FPM</span><span class="o">::</span><span class="no">Cookery</span><span class="o">::</span><span class="no">RubyGemRecipe</span>
<span class="nb">name</span> <span class="s2">"aws-sdk"</span>
<span class="n">version</span> <span class="s2">"1.59.0"</span>
<span class="k">end</span></code></pre></figure>
<p>Next, we pass the recipe to <code class="language-plaintext highlighter-rouge">fpm-cook</code>, the command-line tool that comes with fpm-cookery, and let it build the package for us:</p>
<figure class="highlight"><pre><code class="language-sh" data-lang="sh"><span class="nv">$ </span>fpm-cook package recipe.rb
<span class="o">===></span> Starting package creation <span class="k">for </span>aws-sdk-1.59.0 <span class="o">(</span>debian, deb<span class="o">)</span>
<span class="o">===></span>
<span class="o">===></span> Verifying build_depends and depends with Puppet
<span class="o">===></span> All build_depends and depends packages installed
<span class="o">===></span> <span class="o">[</span>FPM] Trying to download <span class="o">{</span><span class="s2">"gem"</span>:<span class="s2">"aws-sdk"</span>,<span class="s2">"version"</span>:<span class="s2">"1.59.0"</span><span class="o">}</span>
...
<span class="o">===></span> Created package: /home/vagrant/pkg/rubygem-aws-sdk_1.59.0_all.deb</code></pre></figure>
<p>To complete the exercise, we also need to write a recipe for each remaining gem dependency. This is what the final recipes look like:</p>
<figure class="highlight"><pre><code class="language-ruby" data-lang="ruby"><span class="c1"># recipe.rb</span>
<span class="k">class</span> <span class="nc">AwsSdkGem</span> <span class="o"><</span> <span class="no">FPM</span><span class="o">::</span><span class="no">Cookery</span><span class="o">::</span><span class="no">RubyGemRecipe</span>
<span class="nb">name</span> <span class="s2">"aws-sdk"</span>
<span class="n">version</span> <span class="s2">"1.59.0"</span>
<span class="n">maintainer</span> <span class="s2">"Mathias Lafeldt <mathias@example.com>"</span>
<span class="n">chain_package</span> <span class="kp">true</span>
<span class="n">chain_recipes</span> <span class="p">[</span><span class="s2">"aws-sdk-v1"</span><span class="p">,</span> <span class="s2">"json"</span><span class="p">,</span> <span class="s2">"nokogiri"</span><span class="p">]</span>
<span class="k">end</span>
<span class="c1"># aws-sdk-v1.rb</span>
<span class="k">class</span> <span class="nc">AwsSdkV1Gem</span> <span class="o"><</span> <span class="no">FPM</span><span class="o">::</span><span class="no">Cookery</span><span class="o">::</span><span class="no">RubyGemRecipe</span>
<span class="nb">name</span> <span class="s2">"aws-sdk-v1"</span>
<span class="n">version</span> <span class="s2">"1.59.0"</span>
<span class="n">maintainer</span> <span class="s2">"Mathias Lafeldt <mathias@example.com>"</span>
<span class="k">end</span>
<span class="c1"># json.rb</span>
<span class="k">class</span> <span class="nc">JsonGem</span> <span class="o"><</span> <span class="no">FPM</span><span class="o">::</span><span class="no">Cookery</span><span class="o">::</span><span class="no">RubyGemRecipe</span>
<span class="nb">name</span> <span class="s2">"json"</span>
<span class="n">version</span> <span class="s2">"1.8.1"</span>
<span class="n">maintainer</span> <span class="s2">"Mathias Lafeldt <mathias@example.com>"</span>
<span class="k">end</span>
<span class="c1"># nokogiri.rb</span>
<span class="k">class</span> <span class="nc">NokogiriGem</span> <span class="o"><</span> <span class="no">FPM</span><span class="o">::</span><span class="no">Cookery</span><span class="o">::</span><span class="no">RubyGemRecipe</span>
<span class="nb">name</span> <span class="s2">"nokogiri"</span>
<span class="n">version</span> <span class="s2">"1.5.11"</span>
<span class="n">maintainer</span> <span class="s2">"Mathias Lafeldt <mathias@example.com>"</span>
<span class="n">build_depends</span> <span class="p">[</span><span class="s2">"libxml2-dev"</span><span class="p">,</span> <span class="s2">"libxslt1-dev"</span><span class="p">]</span>
<span class="n">depends</span> <span class="p">[</span><span class="s2">"libxml2"</span><span class="p">,</span> <span class="s2">"libxslt1.1"</span><span class="p">]</span>
<span class="k">end</span></code></pre></figure>
<p>Running <code class="language-plaintext highlighter-rouge">fpm-cook</code> again will produce Debian packages that can be added to an APT repository and are ready for use in production.</p>
<p>Three things worth highlighting:</p>
<ul>
<li>
<p>fpm-cookery is able to build multiple dependent packages in a row (configured by <code class="language-plaintext highlighter-rouge">chain_*</code> attributes), allowing us to build everything with a single invocation of <code class="language-plaintext highlighter-rouge">fpm-cook</code>.</p>
</li>
<li>
<p>We can use the attributes <code class="language-plaintext highlighter-rouge">build_depends</code> and <code class="language-plaintext highlighter-rouge">depends</code> to specify a package’s build and runtime dependencies. When running <code class="language-plaintext highlighter-rouge">fpm-cook</code> as root, the tool will automatically install missing dependencies for us.</p>
</li>
<li>
<p>I deliberately set the <code class="language-plaintext highlighter-rouge">maintainer</code> attribute in all recipes. It’s important to take responsibility of the work that we do. We should make it as easy as possible for others to identify the person or team responsible for a package.</p>
</li>
</ul>
<p>fpm-cookery provides <a href="https://github.com/bernd/fpm-cookery/wiki/Recipe-Specification">many more attributes</a> to configure all aspects of the build process. Among other things, it can download source code from GitHub before running custom build instructions (e.g. <code class="language-plaintext highlighter-rouge">make install</code>). The <a href="https://github.com/bernd/fpm-recipes">fpm-recipes</a> repository is an excellent place to study some working examples. This final example, a recipe for <a href="https://github.com/postmodern/chruby">chruby</a>, is a foretaste of what fpm-cookery can actually do:</p>
<figure class="highlight"><pre><code class="language-ruby" data-lang="ruby"><span class="c1"># recipe.rb</span>
<span class="k">class</span> <span class="nc">Chruby</span> <span class="o"><</span> <span class="no">FPM</span><span class="o">::</span><span class="no">Cookery</span><span class="o">::</span><span class="no">Recipe</span>
<span class="n">description</span> <span class="s2">"Changes the current Ruby"</span>
<span class="nb">name</span> <span class="s2">"chruby"</span>
<span class="n">version</span> <span class="s2">"0.3.8"</span>
<span class="n">homepage</span> <span class="s2">"https://github.com/postmodern/chruby"</span>
<span class="n">source</span> <span class="s2">"https://github.com/postmodern/chruby/archive/v</span><span class="si">#{</span><span class="n">version</span><span class="si">}</span><span class="s2">.tar.gz"</span>
<span class="n">sha256</span> <span class="s2">"d980872cf2cd047bc9dba78c4b72684c046e246c0fca5ea6509cae7b1ada63be"</span>
<span class="n">maintainer</span> <span class="s2">"Jan Brauer <jan@example.com>"</span>
<span class="n">section</span> <span class="s2">"development"</span>
<span class="n">config_files</span> <span class="s2">"/etc/profile.d/chruby.sh"</span>
<span class="k">def</span> <span class="nf">build</span>
<span class="c1"># nothing to do here</span>
<span class="k">end</span>
<span class="k">def</span> <span class="nf">install</span>
<span class="n">make</span> <span class="ss">:install</span><span class="p">,</span> <span class="s2">"PREFIX"</span> <span class="o">=></span> <span class="n">prefix</span>
<span class="n">etc</span><span class="p">(</span><span class="s2">"profile.d"</span><span class="p">).</span><span class="nf">install</span> <span class="n">workdir</span><span class="p">(</span><span class="s2">"chruby.sh"</span><span class="p">)</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="c1"># chruby.sh</span>
<span class="n">source</span> <span class="sr">/usr/s</span><span class="n">hare</span><span class="o">/</span><span class="n">chruby</span><span class="o">/</span><span class="n">chruby</span><span class="p">.</span><span class="nf">sh</span></code></pre></figure>
<h2 id="wrapping-up">Wrapping up</h2>
<p>fpm has changed the way we build packages. We can get even more out of fpm by using it in combination with other tools. Dedicated programs like Bundler can help us with resolving package dependencies, which is something fpm won’t do for us. fpm-cookery adds another missing piece: it allows us to describe our packages using simple recipes, which can be kept under version control, giving us the benefits of infrastructure as code: repeatability, automation, rollbacks, code reviews, etc.</p>
<p>Last but not least, it’s a good idea to pair fpm-cookery with <a href="https://www.docker.com/">Docker</a> or <a href="https://www.vagrantup.com/">Vagrant</a> for fast, isolated package builds. This, however, is outside the scope of this article and left as an exercise for the reader.</p>
<h2 id="further-reading">Further reading</h2>
<ul>
<li><a href="http://sysadvent.blogspot.com/2011/12/day-4-guide-to-packaging-systems.html">“A Guide to Package Systems”</a> (SysAdvent 2011, Day 4) – A quick introduction to fpm by its author, <a href="https://twitter.com/jordansissel">Jordan Sissel</a>.</li>
<li><a href="http://sysadvent.blogspot.com/2013/12/day-16-omnibusing-your-way-to-happiness.html">“omnibus’ing your way to happiness”</a> (SysAdvent 2013, Day 16) – An introduction to <a href="https://github.com/opscode/omnibus">Omnibus</a>, a powerful tool that leverages fpm to build full-stack installers.</li>
<li><a href="https://github.com/jordansissel/fpm/wiki">fpm wiki</a> – Comprehensive guides on how to package all sorts of software with fpm.</li>
</ul>
<hr />
<p><em>This article first appeared on the <a href="http://sysadvent.blogspot.com/2014/12/day-15-cook-your-own-packages-getting.html">SysAdvent blog</a> (SysAdvent 2014, Day 15). Thanks to my editor, <a href="http://www.twitter.com/josephkern">Joseph Kern</a>, who did a great job. Also thanks to <a href="http://www.twitter.com/berndahlers">Bernd Ahlers</a>, the author of fpm-cookery, for reviewing drafts of this.</em></p>
Infrastructure automation by example (Practicing Ruby)2013-11-12T00:00:00+01:00https://sharpend.io/infrastructure-automation-by-example