Fast Feedback Is Everything

As an engineer, you should constantly work to make your feedback loops shorter in time and/or wider in scope. – @KentBeck

As software engineers, we spend a lot of our time writing code. Whether we are implementing new features, fixing nasty bugs, or doing boring maintenance work, there is always some code we either create from scratch or try to modify for the better. When developing code, we need certainty that our changes work as intended. That’s why we write tests after (or before) the fact. As a result, a huge part of our daily work inevitably comes down to these steps:

1. Write some code.
2. Write a test that defines the desired behavior.
3. Run the test to see if it’s successful.
4. Go back to 1. or 2. if the test fails. Rinse, repeat.

(For the sake of this article, let’s ignore the tedious test-first vs. test-last vs. TDD debate.)

In order to write an actual program, you have to run through these steps – this loop – again and again. But here’s the thing: constantly jumping back and forth between programming and testing comes at a price. It doesn’t just slow you down in terms of time spent; the involved context switching also drains your mental energy, which might ultimately destroy your productivity.

For this reason, I believe that the following statement is so important – and I’m not tired of repeating it whenever I get the chance:

When it comes to programming and testing, fast feedback is everything.

Fast feedback involves reducing the time between changing code (or tests) and getting test results to a minimum. The faster this feedback loop, the more productive you will be.

There are a couple things you can do to shorten the feedback loop. Certainly the first technique coming to mind is isolated testing, which involves eliminating (slow) external dependencies like databases. Besides trying to implement faster tests, however, you can also optimize the way you run them.

Running tests the fast way

At first glance, how you run tests might not seem to have a big impact on the feedback loop. If a test takes a minute to finish, does it really matter if we can shave off a second or two by tweaking the running step?

Yes, it does. Seconds add up over time. Each additional step requires a little more brain power and incurs a significant context-switching cost.

I don’t like wasting my time with work that can easily be avoided. If there’s a way to minimize the cost of context switching, I’m more than happy to add it to my toolbox. By following the following three steps, I’ve managed to run tests faster and, more importantly, become more productive as a result.

1. Figure out how to execute individual tests. During development, don’t run the entire test suite each time you change a bit of code. Aside from the fact that running all tests is often very slow, it’s always better (and faster!) to first get feedback on local code changes before integrating with other code. Reducing the scope by testing a small subset of code in isolation is not only faster, it also helps you find bugs, and it’s a must-have for TDD. (It goes without saying that you or your continuous integration system should run all the tests at some point.)

2. Write a test runner. This is optional and depends on your test framework/setup. For example, RSpec – one of the best frameworks for testing in Ruby – already allows you to execute a specific test file or even a single test case in that file. Unfortunately, it’s not always that easy. Sometimes you need to execute additional setup/teardown tasks, other times running tests on a package level may be the best you can do. That’s where a test runner comes in handy. In its most basic form, a test runner is a shell script that takes a single argument – the filename of the test you’re currently working on – and does everything required to run the test. I usually store this script as script/test in every project I need it.

3. Run tests using a keyboard shortcut. For fast feedback, it’s important to not leave your editor while hacking on code. Instead, configure your editor of choice to execute tests when pressing a combination of keys on your keyboard. At a minimum, set up a shortcut to run the test currently open in your editor by passing its filename directly to the respective testing tool – or a custom test runner. It’s also useful to have a shortcut for running the test case under the cursor, which will further narrow the focus of your testing.

Examples

Let me give you three practical examples. All of them are somehow related to infrastructure automation, both because it’s an area where rapid feedback matters all the more and because it’s what I do for a living. You will see that I’m a fan of Vim, but it should be straightforward to achieve the same with other editors as well. Here we go:

• rspec-puppet is a test framework that allows to write RSpec tests for Puppet code. When I started working at Jimdo in 2013, it wasn’t possible to run individual tests by simply pointing RSpec at a test file in our codebase. One reason is the unusual way test fixtures are handled in the Puppet world. To remedy this, I wrote a test runner script. Together with vim-spec-runner, a Vim plugin that automatically sets up keyboard shortcuts for running tests, we had everything in place for testing our Puppet code at the whim of a keystroke.

• I primarily developed chef-runner for use with Vim. Instead of jumping back and forth between editing a Chef recipe and running the painfully slow vagrant provision command, I wanted to be able to change code and get immediate feedback without having to leave the editor. chef-runner’s ability to rapidly provision a machine with just a single Chef recipe – the file currently open in Vim – made this possible. There’s no Vim plugin; the setup is as simple as sticking a one-liner in your Vim configuration.

• chef-runner used to be a 100-line shell script before I decided to rewrite it in Go. Go comes with first-class testing support. The go test command is used to run tests (_test.go files) and report test results. However, the tool itself can only run tests for one or more packages based on their import paths; it cannot handle arbitrary _test.go files. These days, I use the :GoTest command from the excellent vim-go plugin to execute package tests for a specific source file.

Wrapping up

Fast feedback plays an important role in software development. Optimizing the way we run tests is one effective method to shorten the feedback loop and get things done.

Sometimes all it takes is a tiny shell script.

Acknowledgment: The ideas presented in this article were heavily inspired by the excellent Destroy All Software screencasts by Gary Bernhardt.