Multi-stage builds #3: Why your build is surprisingly slow, and how to speed it up

If you want your Docker images to be small and you still want fast builds, multi-stage images are the way to go.

And yet, you might find that multi-stage builds are actually quite slow in practice, in particular when running in your build pipeline. If that’s happening, a bit of digging is likely to show that even though you copied your standard build script, somehow the first stage of the Dockerfile gets rebuilt every single time.

Unless you’re very careful, Docker’s build cache often won’t work for multi-stage builds—and that means your build is slow. What’s going on?

In this article you will learn:

  1. Why multi-stage builds don’t work with the standard build process you’re used to for single-stage images.
  2. How to solve the problem and get fast builds.

Note: Outside the specific topic under discussion, the Dockerfiles in this article are not examples of best practices, since the added complexity would obscure the main point of the article.

Want a best-practices Dockerfile and build system? Check out my Production-Ready Python Containers product.

The simple case: Docker caching for normal builds

Consider the following single-stage Dockerfile:

FROM ubuntu:18.04
RUN apt-get update
RUN apt-get install -y --no-install-recommends gcc build-essential

WORKDIR /root
COPY hello.c .
RUN gcc -o helloworld hello.c
CMD ["./helloworld"]

Typically images will be built by a some sort of build or CI system, e.g. Gitlab CI or Circle CI, where the Docker image cache starts out empty.

So unless you’re using a system where the Docker image cache persists across runs, if you want caching to work you will need to pull down the latest image before you do the build:

#!/bin/bash
set -euo pipefail
# Pull the latest version of the image, in order to
# populate the build cache:
docker pull itamarst/helloworld || true
# Build the new version:
docker build -t itamarst/helloworld .
# Push the new version:
docker push itamarst/helloworld

The || true ensures the build will pass the first time you run it, since in that case the pull will fail.

Switching to multi-stage

The Dockerfile we saw above gives you large images, so let’s convert it to a multi-stage build:

# This is the first image:
FROM ubuntu:18.04 AS compile-image
RUN apt-get update
RUN apt-get install -y --no-install-recommends gcc build-essential

WORKDIR /root
COPY hello.c .
RUN gcc -o helloworld hello.c

# This is the second and final image; it copies the compiled binary
# over but starts from the base ubuntu:18.04 image.
FROM ubuntu:18.04 AS runtime-image

COPY --from=compile-image /root/helloworld .
CMD ["./helloworld"]

So far, so good.

However, if you use the same build script as before, caching won’t work. Let’s see why.

$ docker build -q -t twostage .
sha256:e6bc38458e158a3c4aac248434294eede715e0e2e4dade0955d8e7f38d2592bc
$ docker history twostage 
IMAGE               CREATED             CREATED BY                                      SIZE
e6bc38458e15        6 days ago          /bin/sh -c #(nop)  CMD ["./helloworld"]         0B
449dfde8eaf6        6 days ago          /bin/sh -c #(nop) COPY file:e8ef8e1bdaca97ba…   8.3kB
94e814e2efa8        2 weeks ago         /bin/sh -c #(nop)  CMD ["/bin/bash"]            0B    
<missing>           2 weeks ago         /bin/sh -c mkdir -p /run/systemd && echo 'do…   7B    
<missing>           2 weeks ago         /bin/sh -c rm -rf /var/lib/apt/lists/*          0B    
<missing>           2 weeks ago         /bin/sh -c set -xe   && echo '#!/bin/sh' > /…   745B  
<missing>           2 weeks ago         /bin/sh -c #(nop) ADD file:1d7cb45c4e196a6a8…   88.9MB

The history command shows us the layers in the image: we set the CMD, copy the helloworld binary, and then the rest of the layers (the <missing> ones) are from the ubuntu:18.04 image.

Now, imagine we run our normal build, push the resulting image to the image registry at the end of the process:

#!/bin/bash
set -euo pipefail
# Pull the latest version of the image, in order to
# populate the build cache:
docker pull itamarst/helloworld || true
# Build the new version:
docker build -t itamarst/helloworld .
# Push the new version:
docker push itamarst/helloworld

Remember that our assumption is that you’re running in a build environment that starts with an empty cache, an online build service like Circle CI. So next time we do a build, we start with an empty cache, pull the latest image, and—

Where are our layers with apt-get commands? Where is the gcc layer?

They’re not in the cache!

If you only have the final image in your Docker image cache when you do another build, your cache will be missing all of the earlier images, the images you require to do earlier stages of the build. And that means your caching is useless, you will need to rebuild everything from scratch, and your builds will be slow.

The solution: pushing and pulling multiple stages

If you want to have fast builds you need to ensure that the earlier stages—like the stage with the compiler—are also in the Docker image cache. If you’re starting each build with an empty cache, that means you will also need to tag, push, and pull them in the build process—even if you don’t actually use them in production.

#!/bin/bash
set -euo pipefail
# Pull the latest version of the image, in order to
# populate the build cache:
docker pull itamarst/helloworld:compile-stage || true
docker pull itamarst/helloworld:latest        || true

# Build the compile stage:
docker build --target compile-image \
       --cache-from=itamarst/helloworld:compile-stage \
       --tag itamarst/helloworld:compile-stage .

# Build the runtime stage, using cached compile stage:
docker build --target runtime-image \
       --cache-from=itamarst/helloworld:compile-stage \
       --cache-from=itamarst/helloworld:latest \
       --tag itamarst/helloworld:latest .

# Push the new versions:
docker push itamarst/helloworld:compile-stage
docker push itamarst/helloworld:latest

Note: It seems like the --cache-from=itamarst/helloworld:compile-stage --cache-from=itamarst/helloworld:latest is necessary, but possibly not in newest versions of Docker.

The key point is that we explicitly tag and push the compile stage, so that next time we rebuild we can pull it down to populate the cache. Usually the compile stage won’t change, so the only time spent will be downloading the image.

Building Docker images is complex

Docker image builds are the result of interactions between the Dockerfile format, the Docker caching rules, the way Docker images work, and the runtime environment where you do the build. So be careful in assuming that what works in one situation will work in another.

And if you’re using Python, you may be interested in my guide to multi-stage builds for Python applications.


Learn how to build fast, production-ready Docker images—read the rest of the Docker packaging guide for Python.