Using Podman with BuildKit, the better Docker image builder
BuildKit is a new and improved tool for building Docker images: it’s faster, has critical features missing from traditional
Dockerfiles like build secrets, plus additionally useful features like cache mounting.
So if you’re building Docker images, using BuildKit is in general a good idea.
And then there’s Podman: Podman is a reimplemented, compatible version of the Docker CLI and API.
It does not however implement all the BuildKit
On its own, then, Podman isn’t as good as Docker at building images.
There is another option, however: BuildKit has its own build tool, which is distinct from the traditional
docker build, and this build tool can work with Podman.
Let’s see where Podman currently is as far as BuildKit features, and how to use BuildKit with Podman if that is not sufficient.
Build secrets: supported by Podman
Probably the most useful feature added by Buildkit is support for build secrets; standard Docker builds basically had no good way to securely use something like a package repository password.
Dockerfile uses the BuildKit secrets feature:
# syntax = docker/dockerfile:1.3 FROM python:3.9-slim-bullseye RUN --mount=type=secret,id=mysecret echo "Secret is" && cat /run/secrets/mysecret
Note: Outside any specific best practice being demonstrated, the Dockerfiles in this article are not examples of best practices, since the added complexity would obscure the main point of the article.
Make sure your production software is packaged securely, efficiently, and quickly: Read the pragmatic, thorough, and concise Python on Docker Production Handbook.
With normal Docker, we can build it and pass in a secret like so:
$ export DOCKER_BUILDKIT=1 $ echo iamverysecret > secret.txt $ docker build --secret id=mysecret,src=secret.txt --progress=plain . ... #8 [stage-0 2/2] RUN --mount=type=secret,id=mysecret echo "Secret is" && cat /run/secrets/mysecret #8 sha256:78c75c636e72bba25ea3d82e3c7245f68f060a50b40f99be4573db7d2a0318e3 #8 0.361 Secret is #8 0.375 iamverysecret#8 DONE 0.4s ...
As of Podman v3.3, this works with Podman as well:
$ docker build --secret id=mysecret,src=secret.txt . ... STEP 2/2: RUN --mount=type=secret,id=mysecret echo "Secret is" && cat /run/secrets/mysecret WARN Failed to decode the keys ["storage.options.ostree_repo"] from "/home/itamarst/.config/containers/storage.conf". Secret is iam verysecret ...
As of version 3.3, Podman does not yet support the ability to read secrets from environment variables, which is supported by Docker v20.10 and later. But the basic support exists, and that’s good enough.
Caching mounts: not supported by Podman
Another useful BuildKit feature is the ability to cache certain directories across builds, which can for example cache
pip’s download cache, speeding up rebuilds dramatically when dependencies change.
# syntax = docker/dockerfile:1.3 FROM python:3.9-slim-bullseye COPY requirements.txt . RUN --mount=type=cache,target=/root/.cache pip install -r requirements.txt
Unfortunately, Podman does not yet support this:
$ podman build . ... STEP 3/3: RUN --mount=type=cache,target=/root/.cache pip install -r requirements.txt Error: error building at STEP "RUN --mount=type=cache,target=/root/.cache pip install -r requirements.txt": error resolving mountpoints for container "51aeadd15ca2e518ca94a0f866b87a66aea50b5bdc69610bec29fc743338474c": invalid filesystem type "cache" ...
Strictly speaking this feature isn’t necessary, but it is quite nice to have, especially during local development. So how can we access this and other BuildKit features while using Podman?
Using BuildKit with Podman
While BuildKit is built-in to newer versions of Docker, it is also distributed as a separate daemon and command-line tool. And these can run on top of Podman.
Having downloaded the client
buildctl from the link above, we can start the daemon in Podman:
$ podman run -d --name buildkitd --privileged \ docker.io/moby/buildkit:latest
And then we can build images with
$ buildctl --addr=podman-container://buildkitd build \ --frontend dockerfile.v0 \ --local context=. \ --local dockerfile=. \ --export-cache type=inline \ --output type=docker,name=mynewimage | podman load ... Loaded image(s): localhost/latest:latest
If you were to run this with
docker load, you would have an image called
mynewimage visible in
docker image ls.
With Podman v3.3.1, however, the image ends up being called
localhost/latest, which is… not what you’d expect.
localhost part is just Podman’s way of saying “I don’t know what registry this uses”, so that’s fine, but the
latest part is just wrong.
I filed a bug.
The image does get loaded though, albeit with the wrong name, and you can run it with
There are likely other ways to solve this by introducing a third tool, but that seems like even more work; if you know of another solution, please email me and I’ll update the article.
Unlike traditional ways of running image builds, the build cache is not stored in Podman’s image registry, it’s stored by the buildkit daemon, which in this case runs inside another Podman container. So if you restart that daemon the cache goes away, unless you’ve made sure to store it in a volume.
BuildKit can also push newly built images directly to a registry, and can use images in the registry as a source for the cache; see the BuildKit docs for details.
Should you use BuildKit with Podman?
In general, the
buildctl documentation is pretty lacking.
For example, there is no documentation for how you can add labels with
buildctl, though I was able to figure it out by analogy with other commands (
There is also no complete list that I could find of all options.
Possibly it doesn’t exist.
As such, using BuildKit outside of
docker build or the newer
docker buildx can be a frustrating experience.
If you are a Podman user, directly building with Podman supports the most critical feature BuildKit added: build secrets. So unless you need some of BuildKit’s fancier options, at the moment I would suggest just using Podman directly.
The concise and pragmatic guide to Docker packaging for production
Docker packaging for production is complicated, with as many as 70+ best practices to get right. And you want small images, fast builds, and your Python application running securely.
Take the fast path to learning best practices, by using the Python on Docker Production Handbook.
Free ebook: Introduction to Dockerizing for Production
Learn a step-by-step iterative DevOps packaging process in this free mini-ebook. You'll learn what to prioritize, the decisions you need to make, and the ongoing organizational processes you need to start.
Plus, you'll join my newsletter and get weekly articles covering practical tools and techniques, from Docker packaging to Python best practices.