The high cost of slow Docker builds

by
Last updated 01 Oct 2021, originally created 28 Jun 2019

Your Docker build runs too slowly, and it’s annoying—but it’s definitely not an emergency. After all, you’ve got a whole slew of bug fixes and feature requests queued up, and those are more important than your Docker build. So you can never quite justify spending the time to speed it up.

And yes, it’s super-irritating to watch the same 50 Ubuntu packages get downloaded and installed each time over and over and over and over again, and yes, it’s really pointless and wasteful to download and install and compile the same 20 PyPI packages every single time—

—but does it really matter, beyond your personal annoyance?

While it’s true that your Docker image is a means, not an end, a slow build can be very expensive, and well worth your time to speed up. In particular if using the resulting image is part of the critical feedback path for software development, for example if the image is used to run tests, that slowness can significantly impact your team.

A waste of money

Let’s say it takes 6 minutes to run your team’s Docker build. 6 minutes isn’t that long, right?

But if each developer on the team ends up waiting for the Docker build twice a day, and the team has 10 developers, that’s 10×6×2=120 developer minutes/day spent waiting. Put another way, that’s 25% of a single developer’s 8-hour workday.

Given developer salaries in the US, and the fact that a feature that takes a month to develop can be sold and resold to tens or hundreds or thousands of customers, we can reasonably assume that a developer can produce $400K of value in software per year.

Combine those two numbers and we get a sense of the cost of time spent waiting twice a day for a 6-minute Docker build: hundreds of thousands of dollars a year.

  • $100,000/year for 10 developers.
  • $500,000/year for 50 developers
  • $1,000,000/year for 100 developers.

These aren’t accurate numbers, of course, and you may disagree with my assumptions and choose only to count the direct costs, not the opportunity cost. But the orders of magnitude are in the right range: slow Docker builds add up.

A waste of cognitive resources

Beyond a certain point, of course, you aren’t going to sit there staring at you screen waiting for the image build to finish. Instead, you’re going to switch to another task.

Then, when the build finishes (or when you remember to check) you’ll switch back, see if something failed, fix it, rerun the build, and then switch back to the other task.

All this task switching has a cost: remembering what you were doing, why you were doing it, figuring out where your relevant browser tabs are hiding, and so on. On the level of cognitive abilities, there’s extensive psychological research (summarized here by the American Psychological Association) showing that task switching slows down performance of tasks.

Given the tooling and cognitive complexity of programming, the costs of task switching are quite high, though difficult to measure.

Slow builds are expensive!

When your Docker build is a critical bottleneck in the development process, impacting your whole team multiple times a day, small delays can quickly add up to big costs. And these costs are cumulative: there’s the cost of your build, but also the cost of slow tests.

Don’t put off slow Docker builds as an annoyance: do the math on how much time your team is wasting, and then spend a commensurate amount of time speeding it up. A week’s worth of developer time this month will save you a whole lot more over the course of a year.