Docker build secrets: the easy way, the wrong way, the sneaky way
January 2021 update: With the release of a stable BuildKit and Docker 20.10 enabling BuildKit on some environments by default, build secrets should switch to using the relevant BuildKit features. I am leaving this article up for historical purposes.
Building a Docker image often involves installing packages or downloading code, and if you’re installing private code you often need to gain access with a secret: a password, a private key, a token. You don’t want those secrets to end up in the final image, though; if it’s in the image, anyone with access to the image can extract it.
Just to be clear, the goal here is to get build secrets in. Runtime secrets have existing and sufficient mechanisms already (environment variables and volumes).
In this article I’ll cover:
- The eventual, future solution (which you can use now).
- Some broken or problematic solutions you can use now, but probably shouldn’t.
- The sneaky solution: getting secrets in through the network.
Note: Outside the very specific topic under discussion, the Dockerfiles in this article are not examples of best practices, since the added complexity would obscure the main point of the article.
To ensure you’re writing secure, correct, fast Dockerfiles, consider my Python on Docker Production Handbook, which includes a packaging process and >70 best practices.
The future solution: BuildKit
January 2021 update: BuildKit is now stable. You should use it.
Docker is currently working on a new build system called BuildKit, which includes support for adding secrets, as well as for SSH agent authentication forwarding.
BuildKit isn’t enabled by default, but these days it actually seems ready for real-world use.
Just make sure you’re using Docker 19.03 or later, and syntax 1.2 or later, by having
# syntax = docker/dockerfile:1.2 line at the top of your
Nonetheless, if you don’t want to use BuildKit (it still has incompatibilities with Docker Compose, for example), there are other options.
Existing solutions you don’t want to use
Unavailable options: environment variables and volumes
If you’re wondering why not just use environment variables, or volumes, it’s because the goal is to use these secrets as part of
docker build, e.g. to download a package that needs to be installed as part of the build.
docker build doesn’t support environment variables or volumes.
COPY the secret in as a file
If you do this, the secret ends up in one of the image’s layers, even if you delete it in a later layer. Which means an attacker with access to the image can get the secret.
Pass the secret in using –build-arg
Unfortunately build arguments are also embedded in the image: an attacker can run
docker history --no-trunc <yourimage> and see your secrets.
See my article on
docker history for more details.
COPY the secret in to an earlier the stage in a multi-stage build
The idea here is that you do a multi-stage build. In the first stage you install the code using the secret. The second stage copies the resulting code—and then you delete the first stage.
This is secure so long as you’re careful not to push that first image, but this also makes caching much harder, so you’ll end up with slow builds.
The existing solution: the network
Whenever you do a Docker build, your
RUN steps in the Dockerfile have access to the network.
And if you can talk to the network, you can get secrets over the network.
The earliest example I know of that does this is Dockito Vault.
Here I will present a simpler (and simplified) version.
How it works
The method I’m going to use relies on container networking:
- When you run a container, it gets its own network namespace in the kernel, with its own network interfaces and corresponding IP addresses.
- Containers can choose to join the network of an existing container.
docker buildhas a
--networkargument that lets
RUNbuild steps join a particular network—including that of an existing container.
That means we can run a webserver with the secrets, and then the build process can talk to that webserver to get those secrets.
We use the
busybox image to start a container serving a secret, and we call the resulting container
$ cat secret.txt gadzooks123 $ docker run --name=secrets-server --rm --volume $PWD:/files \ busybox httpd -f -p 8000 -h /files
Next, I want to build a
Dockerfile that needs this secret.
For demonstration purposes I will write the secret to stdout, but obviously you don’t want to do this in a real Dockerfile.
FROM busybox RUN echo "The secret is: " && \ wget -O - -q http://localhost:8000/secret.txt
And now we run the build in the network namespace of
secrets-server, so we can access the webserver:
$ docker build --network=container:secrets-server . Sending build context to Docker daemon 3.072kB Step 1/2 : FROM busybox ---> 64f5d945efcc Step 2/2 : RUN echo "The secret is: " && wget -O - -q http://localhost:8000/secret.txt ---> Running in 37db7718316a The secret is: gadzooks123 Removing intermediate container 37db7718316a ---> b464bdd511f9 Successfully built b464bdd511f9
And there you have it: a build that has access to secrets in a secure way.
You will want to make sure you don’t write these secrets to disk beyond the lifetime of a single
RUN line, so that they don’t get persisted in one of the image’s layers.
Writing them to
/dev/shm is a good way to achieve that, since that is an in-memory filesystem and never persisted in the resulting image (nor does it persist for more than the life time of the
Avoid secret leaks
If you have passwords, keys, or tokens you need to build your Docker image, make sure you don’t leak them. Either use the newer BuildKit-based mechanisms, or use the network-based approach I covered here.
Production Docker packaging is too complicated to learn from Google searches
With as much as a dozen different intersecting technologies, and an unknown number of details to get right, Docker packaging isn't simple, especially for production.
But you still need fast builds that save you time, and security best practices that keep you safe.
Take the fast path to learning best practices, by using the Python on Docker Production Handbook.
Learn practical Python software engineering skills you can use at your job
Too much to learn? Don't know where to start?
Sign up for my newsletter, and join over 2600 Python developers and data scientists learning practical tools and techniques, from Docker packaging to testing to Python best practices, with a free new article in your inbox every week.