Multi-stage builds #3: Speeding up your builds

by
Last updated 21 Mar 2023, originally created 01 Apr 2019

If you want your Docker images to be small and you still want fast builds, multi-stage images are the way to go.

And yet, you might find that multi-stage builds are actually quite slow in practice, in particular when running in your build pipeline. If that’s happening, a bit of digging is likely to show that even though you copied your standard build script, somehow the first stage of the Dockerfile gets rebuilt every single time.

Unless you’re very careful, Docker’s build cache often won’t work for multi-stage builds—and that means your build is slow. What’s going on?

In this article you will learn:

  1. Why multi-stage builds don’t work with the standard build process you’re used to for single-stage images.
  2. How to solve the problem and get fast builds.

Note: Outside any specific best practice being demonstrated, the Dockerfiles in this article are not examples of best practices, since the added complexity would obscure the main point of the article.

Python on Docker Production Handbook Need to ship quickly, and don’t have time to figure out every detail on your own? Read the concise, action-oriented Python on Docker Production Handbook.

The simple case: Docker caching for normal builds

Consider the following single-stage Dockerfile; note that 18.04 has hit end-of-life since I first wrote this article, so you should really be using a newer version like 22.04 in the real world:

FROM ubuntu:18.04
RUN apt-get update
RUN apt-get install -y --no-install-recommends gcc build-essential

WORKDIR /root
COPY hello.c .
RUN gcc -o helloworld hello.c
CMD ["./helloworld"]

Typically images will be built by a some sort of build or CI system, e.g. Gitlab CI or Circle CI, where the Docker image cache starts out empty.

So unless you’re using a system where the Docker image cache persists across runs, if you want caching to work you will need to pull down the latest image before you do the build:

#!/bin/bash
set -euo pipefail
# Pull the latest version of the image, in order to
# populate the build cache:
docker pull itamarst/helloworld || true
# Build the new version:
docker build -t itamarst/helloworld .
# Push the new version:
docker push itamarst/helloworld

The || true ensures the build will pass the first time you run it, since in that case the pull will fail.

Switching to multi-stage

The Dockerfile we saw above gives you large images, so let’s convert it to a multi-stage build:

# This is the first image:
FROM ubuntu:18.04 AS compile-image
RUN apt-get update
RUN apt-get install -y --no-install-recommends gcc build-essential

WORKDIR /root
COPY hello.c .
RUN gcc -o helloworld hello.c

# This is the second and final image; it copies the compiled binary
# over but starts from the base ubuntu:18.04 image.
FROM ubuntu:18.04 AS runtime-image

COPY --from=compile-image /root/helloworld .
CMD ["./helloworld"]

So far, so good.

However, if you use the same build script as before, caching won’t work. Let’s see why.

$ docker build -q -t twostage .
sha256:e6bc38458e158a3c4aac248434294eede715e0e2e4dade0955d8e7f38d2592bc
$ docker history twostage 
IMAGE               CREATED             CREATED BY                                      SIZE
e6bc38458e15        6 days ago          /bin/sh -c #(nop)  CMD ["./helloworld"]         0B
449dfde8eaf6        6 days ago          /bin/sh -c #(nop) COPY file:e8ef8e1bdaca97ba…   8.3kB
94e814e2efa8        2 weeks ago         /bin/sh -c #(nop)  CMD ["/bin/bash"]            0B    
<missing>           2 weeks ago         /bin/sh -c mkdir -p /run/systemd && echo 'do…   7B    
<missing>           2 weeks ago         /bin/sh -c rm -rf /var/lib/apt/lists/*          0B    
<missing>           2 weeks ago         /bin/sh -c set -xe   && echo '#!/bin/sh' > /…   745B  
<missing>           2 weeks ago         /bin/sh -c #(nop) ADD file:1d7cb45c4e196a6a8…   88.9MB

The history command shows us the layers in the image: we set the CMD, copy the helloworld binary, and then the rest of the layers (the <missing> ones) are from the ubuntu:18.04 image.

Now, imagine we run our normal build, push the resulting image to the image registry at the end of the process:

#!/bin/bash
set -euo pipefail
# Pull the latest version of the image, in order to
# populate the build cache:
docker pull itamarst/helloworld || true
# Build the new version:
docker build -t itamarst/helloworld .
# Push the new version:
docker push itamarst/helloworld

Remember that our assumption is that you’re running in a build environment that starts with an empty cache, an online build service like Circle CI. So next time we do a build, we start with an empty cache, pull the latest image, and—

Where are our layers with apt-get commands? Where is the gcc layer?

They’re not in the cache!

If you only have the final image in your Docker image cache when you do another build, your cache will be missing all of the earlier images, the images you require to do earlier stages of the build. And that means your caching is useless, you will need to rebuild everything from scratch, and your builds will be slow.

The solution: pushing and pulling multiple stages

If you want to have fast builds you need to ensure that the earlier stages—like the stage with the compiler—are also in the Docker image cache. If you’re starting each build with an empty cache, that means you will also need to tag, push, and pull them in the build process—even if you don’t actually use them in production.

#!/bin/bash
set -euo pipefail
# Pull the latest version of the image, in order to
# populate the build cache. This isn't necessary if 
# you're using BuildKit.
docker pull itamarst/helloworld:compile-stage || true
docker pull itamarst/helloworld:latest        || true

# Build the compile stage:
docker build --target compile-image \
       --build-arg BUILDKIT_INLINE_CACHE=1 \ 
       --cache-from=itamarst/helloworld:compile-stage \
       --tag itamarst/helloworld:compile-stage .

# Build the runtime stage, using cached compile stage:
docker build --target runtime-image \
       --build-arg BUILDKIT_INLINE_CACHE=1 \ 
       --cache-from=itamarst/helloworld:compile-stage \
       --cache-from=itamarst/helloworld:latest \
       --tag itamarst/helloworld:latest .

# Push the new versions:
docker push itamarst/helloworld:compile-stage
docker push itamarst/helloworld:latest

The docker pull makes sure the layers are available locallly–you can skip this if you’re using BuildKit. The --cache-from tells Docker where to get cached layers from, and the BUILDKIT_INLINE_CACHE tells BuildKit to match classic Docker’s behavior and store caching metadata inside the image.

The key point is that we explicitly tag and push the compile stage, so that next time we rebuild we can populate the build cache. Usually the compile stage won’t change, so the only time spent will be downloading the image. If the compile stage is never pushed, it will have to be rebuilt every time.

Building Docker images is complex

Docker image builds are the result of interactions between the Dockerfile format, the Docker caching rules, the way Docker images work, and the runtime environment where you do the build. So be careful in assuming that what works in one situation will work in another.

And if you’re using Python, you may be interested in my guide to multi-stage builds for Python applications.