Don’t leak your Docker image’s build secrets
Building a Docker image often involves installing packages or downloading code, and if you’re installing private code you often need to gain access with a secret: a password, a private key, a token. You don’t want those secrets to end up in the final image, though; if it’s in the image, anyone with access to the image can extract it.
docker run, which supports environment variables (
-e) and volumes,
docker build has traditionally never had a good solution for securely using secrets.
So how do you use build secrets in Docker without leaking them?
In this article you’ll learn:
- Some seemingly reasonable but actually insecure or problematic solutions.
- The easy solution, if you can use modern Docker features.
- The sneaky, backwards-compatible solution: getting secrets in through the network.
- Other potential approaches.
(Preventing leaking other kinds secrets, like runtime secrets, is covered in a different article.)
Insecure options you should not use
Some seemingly reasonable approaches will actually result in the secret (a password, your SSH key) being embedded in the image. That means any attacker getting access to the image will be able to extract your secret.
COPY the secret in as a file
Let’s say you have a
.netrc file with usernames and passwords for your package repository.
It can be tempting to do something like the following:
FROM python:3.9 # Copy in config file with credentials. # DO NOT DO THIS IT IS INSECURE. COPY .netrc /root # pip will use credentials from the .netrc file: RUN pip install https://passwordprotected.repo.example.com/packages/mypackage # Hide the file with credentials: RUN rm /root/.netrc
Don’t do this! Deleting a file does not actually remove it from the image, because Docker uses layer caching: all previous layers are still present in the image. That means the secret ends up in one of the image’s layers, even if you delete it in a later layer.
Any attacker with access to the image can retrieve the secret.
Insecure solution: Pass the secret in using –build-arg
Another tempting approach is to use the
--build-arg option to
Unfortunately build arguments are also embedded in the image: an attacker can run
docker history --no-trunc <yourimage> and see your secrets.
See my article on
docker history for more details.
Technically you can work around this leak by using multi-stage builds, but that will result in slow builds, so I don’t recommend it.
The easy solution: BuildKit
The latest versions of Docker support a new build system called BuildKit, which includes support for adding secrets, as well as for SSH agent authentication forwarding.
Let’s say you have a secret you need to use in your build:
$ cat secret-file THIS IS SECRET
First, configure your
Dockerfile to use BuildKit, and add a flag to
RUN telling it to expose a particular secret:
# syntax = docker/dockerfile:1.2 FROM python:3.8-slim-buster COPY build-script.sh . RUN --mount=type=secret,id=mysecret ./build-script.sh
build-script.sh will be able to find the secret as a file in path
Then, to build your image with the secret set the appropriate environment variable and pass in the newly enabled command-line arguments:
$ export DOCKER_BUILDKIT=1 $ docker build --secret id=mysecret,src=secret-file .
Docker 20.10 adds the additional ability to load secrets from environment variables, not just files.
For example, if you have an environment variable
MYSECRET, you can access it like this:
$ export MYSECRET=theverysecretpassword $ export DOCKER_BUILDKIT=1 $ docker build --secret id=mysecret,env=MYSECRET .
If you’re OK with the id and env variable name being the same, you can replace the last line with:
$ docker build --secret id=MYSECRET .
Note that it will still be exposed inside the build as a file in
/run/secrets, it is merely read from an environment variable on the host.
- BuildKit also support SSH agent forwarding, so you can also access SSH private keys on the host from inside the container without copying any files.
- If you’re using Docker Compose, see this article on using BuildKit build secrets with Compose.
Note: Outside the very specific topic under discussion, the Dockerfiles in this article are not examples of best practices, since the added complexity would obscure the main point of the article.
To ensure you’re following all the best practices you need to have a secure, correct, fast Dockerfiles, check out the Python on Docker Production Handbook.
The backwards-compatible solution: using the network
If you don’t want to use BuildKit, for example if you’re using Podman, there are other options.
Whenever you do a Docker build, your
RUN steps in the Dockerfile have access to the network.
And if you can talk to the network, you can get secrets over the network.
The earliest example I know of that does this is Dockito Vault.
Here I will present a simpler (and simplified) version.
How it works
The method I’m going to use relies on container networking:
- When you run a container, it gets its own network namespace in the kernel, with its own network interfaces and corresponding IP addresses.
- Containers can choose to join the network of an existing container.
docker buildhas a
--networkargument that lets
RUNbuild steps join a particular network—including that of an existing container.
That means we can run a webserver with the secrets, and then the build process can talk to that webserver to get those secrets.
We use the
busybox image to start a container serving a secret, and we call the resulting container
$ cat secret.txt gadzooks123 $ docker run --name=secrets-server --rm --volume $PWD:/files \ busybox httpd -f -p 8000 -h /files
Next, I want to build a
Dockerfile that needs this secret.
For demonstration purposes I will write the secret to stdout, but obviously you don’t want to do this in a real Dockerfile.
FROM busybox RUN echo "The secret is: " && \ wget -O - -q http://localhost:8000/secret.txt
And now we run the build in the network namespace of
secrets-server, so we can access the webserver:
$ docker build --network=container:secrets-server . Sending build context to Docker daemon 3.072kB Step 1/2 : FROM busybox ---> 64f5d945efcc Step 2/2 : RUN echo "The secret is: " && wget -O - -q http://localhost:8000/secret.txt ---> Running in 37db7718316a The secret is: gadzooks123 Removing intermediate container 37db7718316a ---> b464bdd511f9 Successfully built b464bdd511f9
And there you have it: a build that has access to secrets in a secure way.
You will want to make sure you don’t write these secrets to disk beyond the lifetime of a single
RUN line, so that they don’t get persisted in one of the image’s layers.
Writing them to
/dev/shm is a good way to achieve that, since that is an in-memory filesystem and never persisted in the resulting image (nor does it persist for more than the life time of the
Use the build secrets outside of Docker
You don’t necessarily have to use build secrets inside the Docker build, i.e. inside the
You might be able to download all files outside of the Docker build as part of the driving build process, much like you usually check out your code outside of the
Once you have the files downloaded, you can just
COPY them in as usual, and the Docker build never sees the secrets.
Another approach you can take is using expiring tokens, which are supported by some package repositories. Here you login to the package repository outside of the Docker build, and get an access token that will expire in 5 minutes.
You then can pass this temporary token in to the Docker build using
--build-arg, and then use it inside the build process.
The secret will get leaked, it’s true, but since it expires after 5 minutes, as long as you ensure no one can access your image until that deadline is hit, leaking the token isn’t a problem.
Don’t leak those secrets!
If you’re using build secrets, now’s a good time to go and check you’re not leaking them via
You don’t want to find out the hard way that you’ve leaked a secret to an attacker.
Learn how to build fast, production-ready Docker images—read the rest of the Docker packaging guide for Python.
Production Docker packaging is too complicated to learn from Google searches
With as much as a dozen different intersecting technologies, and an unknown number of details to get right, Docker packaging isn't simple, especially for production.
But you still need fast builds that save you time, and security best practices that keep you safe.
Take the fast path to learning best practices, by using the Python on Docker Production Handbook.
Free ebook: Introduction to Dockerizing for Production
Learn a step-by-step iterative DevOps packaging process in this free mini-ebook. You'll learn what to prioritize, the decisions you need to make, and the ongoing organizational processes you need to start.
Plus, you'll join my newsletter and get weekly articles covering practical tools and techniques, from Docker packaging to Python best practices.