Docker builds in CircleCI: go faster, and support newer Linux versions
If you’re using CircleCI to build your Docker images, you might find yourself using an old version of Docker without realizing it. That means:
- Slower builds.
- Lack of support for newer Linux distributions.
Let’s see why, and how to fix it.
BuildKit makes Docker builds faster
Newer versions of Docker add support for BuildKit, a new build backend that among other features (build secrets, and local caching which can speed up builds during development) also can make your production builds faster. In particular, if you’re using multi-stage builds it will try to build the different stages in parallel.
Consider the following multi-stage
FROM ubuntu:20.04 AS build RUN apt-get install -y gcc COPY server.c . RUN gcc server.c -o server FROM ubuntu:20.04 AS runtime # Install security updates: RUN apt-get update && apt-get -y upgrade # Copy executable from build stage: COPY --from=build server . ENTRYPOINT ["./server"]
Note: Outside any specific best practice being demonstrated, the Dockerfiles in this article are not examples of best practices, since the added complexity would obscure the main point of the article.
Need to ship quickly, and don’t have time to figure out every detail on your own? Read the concise, action-oriented Python on Docker Production Handbook.
If I build this both with and without BuildKit, the BuildKit run is much faster, because it can run the two
apt-get update commands in parallel:
$ time docker build --no-cache --quiet . sha256:78a041d11319dc6db92b7c0cf42b00cc8448672e7e4317a79a9dda2b31cd1812 real 0m27.750s user 0m0.016s sys 0m0.017s $ time DOCKER_BUILDKIT=1 docker build --no-cache --quiet . sha256:e094066d3b10de2b28ecb543589ab2f715f47b94f1239ec67a00093fd48b05b6 real 0m17.131s user 0m0.026s sys 0m0.022s
Problem #1: BuildKit not working in CircleCI
Let’s build this image in CircleCI.
First we’ll just configure it normally in the
version: 2.1 jobs: docker-image: docker: - image: cimg/base:stable steps: - checkout - setup_remote_docker - run: | docker build . workflows: docker-image-workflow: jobs: - docker-image
Running that takes 31 seconds.
Next we’ll add the
DOCKER_BUILDKIT environment variable:
# ... - run: | export DOCKER_BUILDKIT=1 docker build . # ...
We run that, and… the build fails:
#!/bin/bash -eo pipefail export DOCKER_BUILDKIT=1 docker build . buildkit not supported by daemon Exited with code exit status 1 CircleCI received exit code 1
What’s going on? A quick look at the previous step in CircleCI gives us a hint:
Server Engine Details: Version: 17.09.0-ce
Apparently CircleCI is running a version of Docker from 2017.
Problem #2: Modern Linux distributions
Even if you don’t care about BuildKit, using old versions of Docker is still a problem. Let’s say you want to build an image with Fedora 35, because you’re testing some software and want to make sure it runs on newer Linux distributions:
FROM fedora:35 RUN dnf install -y python3 # ...
If you try to build this on CircleCI with a Docker version from 2017, you’ll get the following error:
Step 2/2 : RUN dnf install -y python3 ---> Running in 554478328b2f Fedora 35 - x86_64 0.0 B/s | 0 B 00:00 Errors during downloading metadata for repository 'fedora': - Curl error (6): Couldn't resolve host name for https://mirrors.fedoraproject.org/metalink?repo=fedora-35&arch=x86_64&countme=1 [getaddrinfo() thread failed to start] - Curl error (6): Couldn't resolve host name for https://mirrors.fedoraproject.org/metalink?repo=fedora-35&arch=x86_64 [getaddrinfo() thread failed to start] Error: Failed to download metadata for repo 'fedora': Cannot prepare internal mirrorlist: Curl error (6): Couldn't resolve host name for https://mirrors.fedoraproject.org/metalink?repo=fedora-35&arch=x86_64 [getaddrinfo() thread failed to start] The command '/bin/sh -c dnf install -y python3' returned a non-zero code: 1
In fact, you’ll get the same error if you using Docker 19.03, the version they use in most of their examples. The problem is a little complicated (and thanks to Pascal Roeleven for writing the blog post that originally helped me fix this problem when I encountered it) but the short version is that it’s fixed in Docker 20.10.10, released in late October 2021.
Using newer Docker versions in CircleCI
Whether it’s faster builds or support for newer Linux distributions, you need to explicitly opt-in to a new version of Docker when you’re using CircleCI. Otherwise you’ll be stuck with a version from 2017 (by default) or 2019 (if you follow the examples in their documentation).
Here’s how to do it in your
jobs: docker-image: docker: - image: cimg/base:stable steps: - checkout - setup_remote_docker: version: "20.10.11" # <-- modern Docker version!! docker_layer_caching: true - run: | export DOCKER_BUILDKIT=1 docker build .
In addition to specifying the latest Docker version available on CircleCI, we also enabled layer caching, which will cache layers across builds and can add an extra speedup.
Once we’re using a modern Docker, the builds run in just 22 seconds thanks to BuildKit, and the Fedora-based image builds successfully.
Another reason to read the docs
Beyond the CircleCI-specifics, it’s also worth noting that there are a variety of policies a service provider like a CI service might use for upgrading dependencies:
- Use the latest version.
- Upgrade occasionally.
- Stick to an old version for years on end.
Each of these policies has its own tradeoffs, and different impacts on you as a user. So when you start using a service, it’s worth skimming the documentation to see how versions are selected and upgraded.
The concise and action-oriented guide to Docker packaging for production
Docker packaging for production is complicated, with as many as 70+ best practices to get right. And you want small images, fast builds, and your Python application running securely.
Take the fast path to learning best practices, by using the Python on Docker Production Handbook.
Free ebook: Introduction to Dockerizing for Production
Learn a step-by-step iterative DevOps packaging process in this free mini-ebook. You'll learn what to prioritize, the decisions you need to make, and the ongoing organizational processes you need to start.
Plus, you'll join my newsletter and get weekly articles covering practical tools and techniques, from Docker packaging to Python best practices.