How to (not) use Docker to share your password with hackers
Do you use Docker images to run your software? Does running or building your image involve a password or other credential that you really (don’t) want to share with hackers?
Well, you’re in luck, because Docker makes it really easy to share your passwords, cloud credentials, and SSH private keys with the world. Whether it’s runtime secrets, build secrets, or just some random unrelated credentials you had lying around in the wrong place, Docker’s got you covered when it comes to secret leaks.
In this article we’ll cover:
- Some evidence this actually happens.
- Leaking build time secrets.
- Accidental leaks with
- Leaking runtime secrets.
- Some (partially?) missing tooling that would help fix the problem.
Is this really a problem?
Yes, this is a problem.
A high-profile example is CodeCov:
On Thursday, April 1, 2021, we learned that someone had gained unauthorized access to our Bash Uploader script and modified it without our permission. The actor gained access because of an error in Codecov’s Docker image creation process that allowed the actor to extract the credential required to modify our Bash Uploader script.
Among other affected users, HashiCorp had to revoke their GPG signing key (it’s unclear whether or not an attacker actually downloaded it, but they might have):
HashiCorp was impacted by a security incident with a third party (Codecov) that led to potential disclosure of sensitive information. As a result, the GPG key used for release signing and verification has been rotated.
But it’s not just CodeCov. One pentester discovered hundreds of leaked secrets on public Docker images:
I found a wide variety of secrets, such as 200+ AWS accounts (of which 64 were still alive and 14 of these were root), 1,500+ valid SSH keys, Azure keys, several databases, .npmrc tokens, Docker Hub accounts, PyPI repository keys, many SMTP servers, reCAPTCHA secrets, Twitter API keys, Jira keys, Slack keys, and a few others.
As we’ll see later on, the methodology they used likely undercounted secret leaks!
There are multiple ways secrets can get leaked, and it’s unclear which occurred in these case. So let’s go over all of them.
Consider the following common way of copying files into your image:
FROM debian:buster COPY . /app
By default this will copy everything in the current directory into the image.
That everything might include a variety of secrets you didn’t intend to copy in: a
.env file with password, for example.
Deleting the file afterwards with
RUN rm mysecret won’t help, because of the way Docker layers work.
You need to make sure they never get copied in in the first place.
Once the secret is in the Docker image, anyone who can access that image now has access to the secret. This is extra fun when the image is publicly available, but can still hurt you if the image is in a private repository that gets breached.
How can you prevent copying in secrets by mistake?
- Limited copying: Instead of
COPY . /appyou might copy only specific files or directories you know you need. For example,
COPY setup.py myapp /app.
- .dockerignore: You can make sure files don’t get
COPYed in by adding them to the
- Avoid manually building images: Your development machine is much more likely to have random files lying around than an automated build system, so building public images on your dev machine is more likely to leak files.
- Store CI secrets as environment variables: If your CI or build environment needs to use secrets, keep them in environment variables rather than files on disk.
Sometimes you need to use a secret as part of building the Docker image, for example the password to your private package repository.
FROM python:3.9 # How do you get the password in? RUN pip install \ --extra-index-url https://firstname.lastname@example.org \ yourpackage anotherprivatepackage
The obvious ways to get these passwords in, like build arguments and
COPY, will result in the secret being embedded in the image.
There are a variety of techniques that are secure, which I cover in a separate article specifically about Docker build secrets.
Sometimes you need some secret to actually run your software in production. For example, your web application might require the password to its MySQL database. These are known as runtime secrets, and the important thing to remember is that, once again, you should not store them in your image.
Once they’re in the image, any attacker who gets access to the image can get that secret, e.g. the MySQL password. Plus, it means your image is tied to a specific runtime environment, and it’ll be harder to take your image and run it somewhere else.
Instead, you can pass secrets into your container when it’s run, via a variety of methods:
- Using environment variables.
- Bind mounting a volume with the secret.
- Secret-specific mechanisms like Kubernetes secrets (which actually uses both mechanisms above).
- In some cloud environments, you can give permissions to containers to talk to the cloud environment, e.g. IAM roles for ECS tasks on AWS.
- Retrieving them from some sort of external key store.
With all of these mechanisms the secret is never stored in the image itself.
Finding leaks with scanners
Beyond the guidelines given above, it’s also worth scanning for secrets that accidentally made it through anyway.
There are a variety of tools like
trufflehog, and more that will scan a directory or Git repository’s history for things that looked like leaked secrest.
Unfortunately I have not found many tools that do a good job for Docker images. For example, the pentester who scanned Docker Hub images just scanned the filesystem exposed to the container. That will miss secrets that are still in the image, since you also want to scan:
- All layers in the image, to find secrets masked by
RUN rm mysecret.
- The image build history, which can include secrets leaked via build args.
The only tool that seemed like it might work didn’t run out of the box, so I can’t tell whether it actually scans correctly. If you know of tools that actually scan correctly for secrets (i.e. all layers and history), please let me know.
For now, follow the guidelines in the rest of the article, and consider using a tool like
detect-secrets within your container to at least do a naive better-than-nothing secrets scan.
Learn how to build fast, production-ready Docker images—read the rest of the Docker packaging guide for Python.
Production Docker packaging is too complicated to learn from Google searches
With as much as a dozen different intersecting technologies, and an unknown number of details to get right, Docker packaging isn't simple, especially for production.
But you still need fast builds that save you time, and security best practices that keep you safe.
Take the fast path to learning best practices, by using the Python on Docker Production Handbook.
Free ebook: Introduction to Dockerizing for Production
Learn a step-by-step iterative DevOps packaging process in this free mini-ebook. You'll learn what to prioritize, the decisions you need to make, and the ongoing organizational processes you need to start.
Plus, you'll join my newsletter and get weekly articles covering practical tools and techniques, from Docker packaging to Python best practices.