Multi-stage builds #1: Smaller images for compiled code

You’re building a Docker image for a Python project with compiled code (C/C++/Rust/whatever), and somehow without quite realizing it you’ve created a Docker image that is 917MB… only 1MB of which is your code! Bigger images mean slower deploys, slower test runs, potentially higher bandwidth costs: they’re a waste of time and money.

However, with a little work it is possible to have smaller Docker images, even if you need to install a large compile and build toolchain.

In this article I’ll cover:

1. Why the architecture of Docker image layers makes images larger than you’d expect.
2. The single-layer solution, which works but suffers from other performance problems.
3. The better solution: two-stage builds.

Note: Outside any specific best practice being demonstrated, the Dockerfiles in this article are not examples of best practices, since the added complexity would obscure the main point of the article.

Need to ship quickly, and don’t have time to figure out every detail on your own? Read the concise, action-oriented Python on Docker Production Handbook.

An example of an unexpectedly large image

Consider the following simple Docker image (note that since I’ve written this article, there have been enough new Python releases that 3.7 is deprecated, and 3.8 is about to be); you should be using Python 3.12 at least.

FROM python:3.7-slim

RUN apt-get update
RUN apt-get install -y --no-install-recommends gcc

COPY myapp/ .
COPY setup.py .

RUN python setup.py install

We install a compiler, and compile the code—and the resulting image is 243MB in size.

The obvious solution is to uninstall the compiler after compilation is done, but this won’t work:

FROM python:3.7-slim

RUN apt-get update
RUN apt-get install -y --no-install-recommends gcc

COPY myapp/ .
COPY setup.py .
RUN python setup.py install

RUN apt-get remove -y gcc
RUN apt-get -y autoremove

If we build the above, the image size will be… 245MB. Uninstalling all those packages did us no good!

What’s going on?

Problem #1: Docker images are append-only

A Docker image is built as a sequence of layers, with each layer building on top of the previous one. Among other things, layers can add files, or they can remove files.

Removing files doesn’t remove it from the previous layer, and all the layers are needed to unpack the Docker image.

If we simplify the Dockerfile above, essentially we have:

1. Layer A: Add lots of files for the compiler.
2. Layer B(→A): Compile some code.
3. Layer C(→B→A): Remove the files for the compiler

To download the image we need layers A, B, and C, so we have to download the compiler even though the relevant files aren’t accessible in the final image.

Solution #1: A single layer

Each line in a Dockerfile adds a new layer, and the problem is that the compiler’s files are stored indefinitely once the relevant RUN layer finishes running.

This suggests a solution: make sure all the files are gone by the time the RUN finishes.

FROM python:3.7-slim

COPY myapp/ .
COPY setup.py .

RUN apt-get update && \
    apt-get install -y --no-install-recommends gcc && \
    python setup.py install && \
    apt-get remove -y gcc && apt-get -y autoremove

And indeed, this image is 161MB: we’ve succeeded in removing at least some of the unnecessary files.

This is a little ugly and hard to read, though. Another alternative is to switch back to our nicer original Dockerfile, with file deletes in their own RUN lines, and call docker build --squash. Squashing means combining all the layers into a single layer in the final image. This involves enabling experimental features, but again should give us a smaller image.

Both these variants have a problem, though: we’ve destroyed Docker’s build cache.

Problem #2: Single layer combos make builds slower

Let’s go back to our original, multi-layer Dockerfile:

FROM python:3.7-slim

RUN apt-get update
RUN apt-get install -y --no-install-recommends gcc

COPY myapp/ .
COPY setup.py .
RUN python setup.py install

RUN apt-get remove -y gcc
RUN apt-get -y autoremove

We’ll time building the image from scratch, modify a file, and then rebuild it:

$ time docker build -q -t myapp .
sha256:7f77dbea8e84bf6ac329bb71f77dff17030aa6c760a1e177d7d536b198ba9d78

real    0m56.625s
user    0m0.042s
sys     0m0.023s

$ echo '# modifying this file' >> myapp/__init__.py

$ time docker build -q -t myapp .
sha256:db04943f893395fa49f8a2561a36deabde8a67a3ff5fb21e24b2a4deca95a557

real    0m9.611s
user    0m0.030s
sys     0m0.034s

The second build goes much faster. Why?

Because Docker builds have a cache:

• If the files in a COPY line changes, then that layer is invalidated from the cache and all subsequent layers need to be rebuilt.
• If the command in a RUN line changes, then that layer is invalidated. Again, all subsequent layers need to be rebuilt.
• Otherwise, a layer doesn’t need to rebuilt, and can just be taken from the cache.

In the example above, the apt-get RUN lines haven’t changed, and so they get cached: even though the source code has changed, we don’t need to redownload and reinstall all those packages. The result is a faster build.

If we only have a single layer, however, we lose that benefit. We need to copy the source code in before we run apt-get:

FROM python:3.7-slim

COPY myapp/ .
COPY setup.py .

RUN apt-get update && \
    apt-get install -y --no-install-recommends gcc && \
    python setup.py install && \
    apt-get remove -y gcc && apt-get -y autoremove

And that means every time we change the source code, all the packages will have be redownloaded and reinstalled. A single layer gives us smaller images, but at the cost of slower builds.

Solution #2: Multi-stage builds

There’s a better solution that can give us both smaller images and faster builds: multi-stage builds.

In multi-stage builds, you build two different images:

1. In the first image, you install the compiler and other build tools, and then compile your source code.
2. In the second image, you install only the packages you need to run your code, and you copy in the compiled artifacts created by the first image.

The second image is what you use to run your code. This process means you can rely on Docker’s caching to speed up builds, while still getting the benefit of smaller image.

So how do you do a multi-stage build? I cover this in two follow-up articles:

1. Writing the Dockerfile and Python specifics: There are different ways you can approach creating and copying the build artifacts, each with its own tradeoffs.
2. Ensuring fast builds: When run in a CI system, naive usage of multi-stage builds will often result in slow builds due to uncached data. Here’s how to get faster multi-stage builds.