Using Podman with BuildKit, the better Docker image builder
BuildKit is a new and improved tool for building Docker images: it’s faster, has critical features missing from traditional
Dockerfiles like build secrets, plus additionally useful features like cache mounting.
So if you’re building Docker images, using BuildKit is in general a good idea.
And then there’s Podman: Podman is a reimplemented, compatible version of the Docker CLI and API.
It did not initially implement all the BuildKit
Dockerfile extensions, but it has been adding more over time.
There is another option, however: BuildKit has its own build tool, which is distinct from the traditional
docker build, and this build tool can work with Podman.
Let’s see where Podman currently is as far as BuildKit features, and how to use BuildKit with Podman if that is not sufficient.
Using Podman’s support for BuildKit features
Over time, Podman has added support for the new
Dockerfile features added by BuildKit.
Probably the most useful feature added by Buildkit is support for build secrets; standard Docker builds basically had no good way to securely use something like a package repository password.
Dockerfile uses the BuildKit secrets feature:
# syntax = docker/dockerfile:1.3 FROM python:3.9-slim-bullseye RUN --mount=type=secret,id=mysecret echo "Secret is" && cat /run/secrets/mysecret
Note: Outside any specific best practice being demonstrated, the Dockerfiles in this article are not examples of best practices, since the added complexity would obscure the main point of the article.
Need to ship quickly, and don’t have time to figure out every detail on your own? Read the concise, action-oriented Python on Docker Production Handbook.
With normal Docker, we can build it and pass in a secret like so:
$ export DOCKER_BUILDKIT=1 $ echo iamverysecret > secret.txt $ docker build --secret id=mysecret,src=secret.txt --progress=plain . ... #8 [stage-0 2/2] RUN --mount=type=secret,id=mysecret echo "Secret is" && cat /run/secrets/mysecret #8 sha256:78c75c636e72bba25ea3d82e3c7245f68f060a50b40f99be4573db7d2a0318e3 #8 0.361 Secret is #8 0.375 iamverysecret#8 DONE 0.4s ...
With Podman v3.3, this works with Podman as well:
$ docker build --secret id=mysecret,src=secret.txt . ... STEP 2/2: RUN --mount=type=secret,id=mysecret echo "Secret is" && cat /run/secrets/mysecret WARN Failed to decode the keys ["storage.options.ostree_repo"] from "/home/itamarst/.config/containers/storage.conf". Secret is iam verysecret ...
Another useful BuildKit feature is the ability to cache certain directories across builds, which can for example cache
pip’s download cache, speeding up rebuilds dramatically when dependencies change.
# syntax = docker/dockerfile:1.3 FROM python:3.9-slim-bullseye COPY requirements.txt . RUN --mount=type=cache,target=/root/.cache pip install -r requirements.txt
Older versions of Podman didn’t support this, but nnewer versions of Podman do; I tested with Podman 4.1.1. In general, Podman is doing a good job adding support for additional BuildKit features.
Using BuildKit with Podman
As an alternative to using Podman’s image building functionality, another approach is to use BuildKit directly. While BuildKit is built-in to newer versions of Docker, it is also distributed as a separate daemon and command-line tool. And these can run on top of Podman.
Having downloaded the client
buildctl from the link above, we can start the daemon in Podman:
$ podman run -d --name buildkitd --privileged \ docker.io/moby/buildkit:latest
And then we can build images with
$ buildctl --addr=podman-container://buildkitd build \ --frontend dockerfile.v0 \ --local context=. \ --local dockerfile=. \ --export-cache type=inline \ --output type=docker,name=mynewimage | podman load ... Loaded image(s): localhost/latest:latest
If you were to run this with
docker load, you would have an image called
mynewimage visible in
docker image ls.
When I first wrote this article, with Podman v3.3.1, however, the image ended up being called
localhost/latest, which is… not what you’d expect.
localhost part is just Podman’s way of saying “I don’t know what registry this uses”, so that’s fine, but the
latest part is just wrong.
I filed a bug and it was fixed.
Testing in July 2022 on Ubuntu 22.04 with newer versions of Podman, I was unable to get this to work at all. Whether this is due to buildkit, Podman, or an interaction between the two is unclear.
Unlike traditional ways of running image builds, the build cache is not stored in Podman’s image registry, it’s stored by the buildkit daemon, which in this case runs inside another Podman container. So if you restart that daemon the cache goes away, unless you’ve made sure to store it in a volume.
BuildKit can also push newly built images directly to a registry, and can use images in the registry as a source for the cache; see the BuildKit docs for details.
Should you use BuildKit with Podman?
In general, the
buildctl documentation is pretty lacking.
For example, there is no documentation for how you can add labels with
buildctl, though I was able to figure it out by analogy with other commands (
There is also no complete list that I could find of all options.
Possibly it doesn’t exist.
As such, using BuildKit outside of
docker build or the newer
docker buildx can be a frustrating experience.
And my attempts to use it with Podman have not been very successful.
If you are a Podman user, directly building with Podman supports a number of BuildKit features (build secrets, mount caches), with more being added over time. So unless you need some of BuildKit’s fancier options, I would suggest just using Podman directly.
The concise and action-oriented guide to Docker packaging for production
Docker packaging for production is complicated, with as many as 70+ best practices to get right. And you want small images, fast builds, and your Python application running securely.
Take the fast path to learning best practices, by using the Python on Docker Production Handbook.
Free ebook: Introduction to Dockerizing for Production
Learn a step-by-step iterative DevOps packaging process in this free mini-ebook. You'll learn what to prioritize, the decisions you need to make, and the ongoing organizational processes you need to start.
Plus, you'll join my newsletter and get weekly articles covering practical tools and techniques, from Docker packaging to Python best practices.