Using Podman with BuildKit, the better Docker image builder

by
Last updated 21 Mar 2023, originally created 16 Sep 2021

BuildKit is a new and improved tool for building Docker images: it’s faster, has critical features missing from traditional Dockerfiles like build secrets, plus additionally useful features like cache mounting. So if you’re building Docker images, using BuildKit is in general a good idea.

And then there’s Podman: Podman is a reimplemented, compatible version of the Docker CLI and API. It did not initially implement all the BuildKit Dockerfile extensions, but it has been adding more over time.

There is another option, however: BuildKit has its own build tool, which is distinct from the traditional docker build, and this build tool can work with Podman.

Let’s see where Podman currently is as far as BuildKit features, and how to use BuildKit with Podman if that is not sufficient.

Using Podman’s support for BuildKit features

Over time, Podman has added support for the new Dockerfile features added by BuildKit.

Build secrets

Probably the most useful feature added by Buildkit is support for build secrets; standard Docker builds basically had no good way to securely use something like a package repository password.

The following Dockerfile uses the BuildKit secrets feature:

# syntax = docker/dockerfile:1.3
FROM python:3.9-slim-bullseye
RUN --mount=type=secret,id=mysecret echo "Secret is" && cat /run/secrets/mysecret

Note: Outside any specific best practice being demonstrated, the Dockerfiles in this article are not examples of best practices, since the added complexity would obscure the main point of the article.

Python on Docker Production Handbook Need to ship quickly, and don’t have time to figure out every detail on your own? Read the concise, action-oriented Python on Docker Production Handbook.

With normal Docker, we can build it and pass in a secret like so:

$ export DOCKER_BUILDKIT=1
$ echo iamverysecret > secret.txt
$ docker build --secret id=mysecret,src=secret.txt --progress=plain .
...
#8 [stage-0 2/2] RUN --mount=type=secret,id=mysecret echo "Secret is" && cat /run/secrets/mysecret
#8 sha256:78c75c636e72bba25ea3d82e3c7245f68f060a50b40f99be4573db7d2a0318e3
#8 0.361 Secret is
#8 0.375 iamverysecret#8 DONE 0.4s
...

With Podman v3.3, this works with Podman as well:

$ docker build --secret id=mysecret,src=secret.txt .
...
STEP 2/2: RUN --mount=type=secret,id=mysecret echo "Secret is" && cat /run/secrets/mysecret
WARN[0000] Failed to decode the keys ["storage.options.ostree_repo"] from "/home/itamarst/.config/containers/storage.conf". 
Secret is
iam verysecret
...

Caching mounts

Another useful BuildKit feature is the ability to cache certain directories across builds, which can for example cache pip’s download cache, speeding up rebuilds dramatically when dependencies change.

# syntax = docker/dockerfile:1.3
FROM python:3.9-slim-bullseye
COPY requirements.txt .
RUN --mount=type=cache,target=/root/.cache pip install -r requirements.txt

Older versions of Podman didn’t support this, but newer versions of Podman do; I tested with Podman 4.1.1. In general, Podman is doing a good job adding support for additional BuildKit features.

Using BuildKit with Podman

As an alternative to using Podman’s image building functionality, another approach is to use BuildKit directly. While BuildKit is built-in to newer versions of Docker, it is also distributed as a separate daemon and command-line tool. And these can run on top of Podman.

Having downloaded the client buildctl from the link above, we can start the daemon in Podman:

$ podman run -d --name buildkitd --privileged \
     docker.io/moby/buildkit:latest

And then we can build images with buildctl:

$ buildctl --addr=podman-container://buildkitd build \
   --frontend dockerfile.v0 \
   --local context=. \
   --local dockerfile=. \
   --export-cache type=inline \
   --output type=docker,name=mynewimage | podman load
...
Loaded image(s): localhost/latest:latest

If you were to run this with docker load, you would have an image called mynewimage visible in docker image ls. When I first wrote this article, with Podman v3.3.1, however, the image ended up being called localhost/latest, which is… not what you’d expect. The localhost part is just Podman’s way of saying “I don’t know what registry this uses”, so that’s fine, but the latest part is just wrong. I filed a bug and it was fixed.

Testing in July 2022 on Ubuntu 22.04 with newer versions of Podman, I was unable to get this to work at all. Whether this is due to buildkit, Podman, or an interaction between the two is unclear.

Unlike traditional ways of running image builds, the build cache is not stored in Podman’s image registry, it’s stored by the buildkit daemon, which in this case runs inside another Podman container. So if you restart that daemon the cache goes away, unless you’ve made sure to store it in a volume.

BuildKit can also push newly built images directly to a registry, and can use images in the registry as a source for the cache; see the BuildKit docs for details.

Should you use BuildKit with Podman?

In general, the buildctl documentation is pretty lacking (at least when I wrote this article in 2021, in the interim things might have improved.) For example, there is no documentation for how you can add labels with buildctl, though I was able to figure it out by analogy with other commands (--opt label:labelname=labelvalue). There is also no complete list that I could find of all options. Possibly it doesn’t exist.

As such, using BuildKit outside of docker build or the newer docker buildx can be a frustrating experience. And my attempts to use it with Podman have not been very successful.

If you are a Podman user, directly building with Podman supports a number of BuildKit features (build secrets, mount caches), with more being added over time. So unless you need some of BuildKit’s fancier options, I would suggest just using Podman directly.