Docker builds in CircleCI: go faster, and support newer Linux versions

by
Last updated 19 Sep 2022, originally created 28 Feb 2022

Important: CircleCI has since updated the default version of Docker to a much more reasonable version, 20.10. So you don’t have to configure anything yourself anymore, and while this article might still teach you something, it is mostly obsolete.

If you’re using CircleCI to build your Docker images, you might find yourself using an old version of Docker without realizing it. That means:

  • Slower builds.
  • Lack of support for newer Linux distributions.

Let’s see why, and how to fix it.

BuildKit makes Docker builds faster

Newer versions of Docker add support for BuildKit, a new build backend that among other features (build secrets, and local caching which can speed up builds during development) also can make your production builds faster. In particular, if you’re using multi-stage builds it will try to build the different stages in parallel.

Consider the following multi-stage Dockerfile:

FROM ubuntu:20.04 AS build
RUN apt-get install -y gcc
COPY server.c .
RUN gcc server.c -o server

FROM ubuntu:20.04 AS runtime
# Install security updates:
RUN apt-get update && apt-get -y upgrade
# Copy executable from build stage:
COPY --from=build server .
ENTRYPOINT ["./server"]

Note: Outside any specific best practice being demonstrated, the Dockerfiles in this article are not examples of best practices, since the added complexity would obscure the main point of the article.

Python on Docker Production Handbook Need to ship quickly, and don’t have time to figure out every detail on your own? Read the concise, action-oriented Python on Docker Production Handbook.

If I build this both with and without BuildKit, the BuildKit run is much faster, because it can run the two apt-get update commands in parallel:

$ time docker build --no-cache --quiet .
sha256:78a041d11319dc6db92b7c0cf42b00cc8448672e7e4317a79a9dda2b31cd1812

real    0m27.750s
user    0m0.016s
sys     0m0.017s
$ time DOCKER_BUILDKIT=1 docker build --no-cache --quiet .
sha256:e094066d3b10de2b28ecb543589ab2f715f47b94f1239ec67a00093fd48b05b6

real    0m17.131s
user    0m0.026s
sys     0m0.022s

Problem #1: BuildKit not working in CircleCI

Let’s build this image in CircleCI. First we’ll just configure it normally in the .circleci/config.yml:

version: 2.1

jobs:
  docker-image:
    docker:
      - image: cimg/base:stable
    steps:
      - checkout
      - setup_remote_docker
      - run: |
          docker build .

workflows:
  docker-image-workflow:
    jobs:
      - docker-image

Running that takes 31 seconds. Next we’ll add the DOCKER_BUILDKIT environment variable:

# ...
      - run: |
          export DOCKER_BUILDKIT=1
          docker build .
# ...

We run that, and… the build fails:

#!/bin/bash -eo pipefail
export DOCKER_BUILDKIT=1
docker build .

buildkit not supported by daemon

Exited with code exit status 1

CircleCI received exit code 1

What’s going on? A quick look at the previous step in CircleCI gives us a hint:

Server Engine Details:
  Version:          17.09.0-ce

Apparently CircleCI is running a version of Docker from 2017.

Problem #2: Modern Linux distributions

Even if you don’t care about BuildKit, using old versions of Docker is still a problem. Let’s say you want to build an image with Fedora 35, because you’re testing some software and want to make sure it runs on newer Linux distributions:

FROM fedora:35
RUN dnf install -y python3
# ...

If you try to build this on CircleCI with a Docker version from 2017, you’ll get the following error:

Step 2/2 : RUN dnf install -y python3
 ---> Running in 554478328b2f
Fedora 35 - x86_64                              0.0  B/s |   0  B     00:00    
Errors during downloading metadata for repository 'fedora':
  - Curl error (6): Couldn't resolve host name for https://mirrors.fedoraproject.org/metalink?repo=fedora-35&arch=x86_64&countme=1 [getaddrinfo() thread failed to start]
  - Curl error (6): Couldn't resolve host name for https://mirrors.fedoraproject.org/metalink?repo=fedora-35&arch=x86_64 [getaddrinfo() thread failed to start]
Error: Failed to download metadata for repo 'fedora': Cannot prepare internal mirrorlist: Curl error (6): Couldn't resolve host name for https://mirrors.fedoraproject.org/metalink?repo=fedora-35&arch=x86_64 [getaddrinfo() thread failed to start]
The command '/bin/sh -c dnf install -y python3' returned a non-zero code: 1

In fact, you’ll get the same error if you using Docker 19.03, the version they use in most of their examples. The problem is a little complicated (and thanks to Pascal Roeleven for writing the blog post that originally helped me fix this problem when I encountered it) but the short version is that it’s fixed in Docker 20.10.10, released in late October 2021.

Using newer Docker versions in CircleCI

Whether it’s faster builds or support for newer Linux distributions, you need to explicitly opt-in to a new version of Docker when you’re using CircleCI. Otherwise you’ll be stuck with a version from 2017 (by default) or 2019 (if you follow the examples in their documentation).

Here’s how to do it in your config.yml:

jobs:
  docker-image:
    docker:
      - image: cimg/base:stable
    steps:
      - checkout
      - setup_remote_docker:
          version: "20.10.11"  # <-- modern Docker version!!
          docker_layer_caching: true
      - run: |
          export DOCKER_BUILDKIT=1
          docker build .

In addition to specifying the latest Docker version available on CircleCI, we also enabled layer caching, which will cache layers across builds and can add an extra speedup.

Once we’re using a modern Docker, the builds run in just 22 seconds thanks to BuildKit, and the Fedora-based image builds successfully.

Another reason to read the docs

Beyond the CircleCI-specifics, it’s also worth noting that there are a variety of policies a service provider like a CI service might use for upgrading dependencies:

  • Use the latest version.
  • Upgrade occasionally.
  • Stick to an old version for years on end.

Each of these policies has its own tradeoffs, and different impacts on you as a user. So when you start using a service, it’s worth skimming the documentation to see how versions are selected and upgraded.