Stuck with slow tests? Speed up your feedback loop
You submit a pull request, the CI system kicks off the tests—and then you wait. And wait some more. And by the time it turns red to tell you there’s a problem you’ve moved on to something else.
What were you doing again?
And eventually, hours or even days later, you finally get your code merged.
For a single developer all this waiting and context switching is expensive; for a whole team it’s that much worse. Obviously what you need is a faster test suite.
But what if you can’t make your test suite faster? You don’t have the time, or don’t have the skills, or maybe you’ve optimized it so much you don’t think you can do any better. Are you just stuck wasting your time?
Don’t give up: there is something you can do. What you can do is speed up the feedback loop of your test system.
In this article I’ll explain:
- Why faster feedback is what really matters.
- An overview of some of the ways you can speed up your testing feedback loop.
Why you need a faster feedback loop
When your CI systems runs your tests, there are two possible outcomes:
- Success.
- Failure.
If your tests pass successfully you can probably just merge your code—in fact some software teams will have the branch merge automatically if it’s passed review and tests. There’s not much to do, really, and if it takes a while to get there that’s not ideal, but it’s also not that bad.
If your tests fail after 30 minutes, however, you will need to:
- Stop whatever new task you switched to.
- Try and remember what that failing code did.
- Figure out the problem and fix it.
- Switch back to your new task.
- Remember what you were doing on your new task.
And this may repeat multiple times if your fix was insufficient or problematic and the tests fail again (as they often do).
In short, a failing CI run is a much bigger timesink than a successful run, because of all the expensive mental context switching it causes. In particular, failing tests are expensive when it takes 10 minutes, 30 minutes, or even 2 hours until you know that your code needs to be fixed.
If you can speed up the time-to-reported-failure of your test suite, if you can get meaningful feedback faster, you can reduce or eliminate the mental context switching. If you push code and almost immediately get told your tests are failing, you can just fix them immediately, and repeat until you’re fairly certain that everything is going to pass.
Some ways to speed up your feedback loop
Even if you can’t speed up your test suite overall, then, you can still make failures occur as quickly as possible. It’s much more useful if your CI run fails within 3 minutes than if it fails after 90 minutes.
Here are some of the ways you can do that:
Linters and code analyzers
The first thing your test suite should do is run a linter. The linter can catch obvious problems before the relevant tests do, so if it runs first, fails, and then ends your CI run, you’ll be notified of the problems faster.
Personally I’m a fan of pylint (when it’s configured correctly), but you can also use flake8 or other tools. And mypy or other type checking tools can help catch problems if you’re using type annotations.
You can run these tools quite early—depending on how you’ve configured them, before you’ve installed any dependencies, allowing your CI run to provide feedback even faster.
Run relevant tests first
If you changed module water.py, chances are the tests that will fail are in test_water.py and test_bucket.py, not in test_steel.py.
By running relevant tests first, you can increases the chances of failing quickly.
There are a variety of ways to do this, from manually recording the dependency information (which is how the Twisted project does it) to heuristic tools like py.test-testmon and coverage-based tools like partialtesting.
You should still run the whole test suite, but only after running the relevant tests. If you can’t avoid it, running the same tests twice (once because they are relevant, and then a second time when you run the full test suite) may still be worth it for the faster feedback loop.
Run faster and smaller tests first
If you have two sets of tests, fast small scale tests (one of the usages of “unit test”) and slow integration tests, run the fast tests first. With any luck they’ll catch the problem before you get to the slow integration tests.
Speed up the rest of your development process
Tests are just part of your development process, and you might be able to speed up the feedback loop elsewhere as well. For example, if you currently do code reviews after tests pass, maybe instead you want to do them immediately when the pull request is submitted, in parallel with CI.
Faster feedback, faster development
The suggestions above are just the beginning; likely there are others way you can optimize for feedback in your particular situation. Just remember: it’s the speed of feedback that matters, and the easiest way to speed up feedback is to have your test suite find relevant failures as quickly as possible.
The faster your feedback loop, the less need there is for context switching—and the faster you’ll be able to ship features and bug fixes.
