How to (not) use Docker to share your password with hackers

by
Last updated 02 Feb 2022, originally created 05 May 2021

Do you use Docker images to run your software? Does running or building your image involve a password or other credential that you really (don’t) want to share with hackers?

Well, you’re in luck, because Docker makes it really easy to share your passwords, cloud credentials, and SSH private keys with the world. Whether it’s runtime secrets, build secrets, or just some random unrelated credentials you had lying around in the wrong place, Docker’s got you covered when it comes to secret leaks.

In this article we’ll cover:

  • Some evidence this actually happens.
  • Leaking build time secrets.
  • Accidental leaks with COPY.
  • Leaking runtime secrets.
  • Some (partially?) missing tooling that would help fix the problem.

Is this really a problem?

Yes, this is a problem.

A high-profile example is CodeCov:

On Thursday, April 1, 2021, we learned that someone had gained unauthorized access to our Bash Uploader script and modified it without our permission. The actor gained access because of an error in Codecov’s Docker image creation process that allowed the actor to extract the credential required to modify our Bash Uploader script.

Among other affected users, HashiCorp had to revoke their GPG signing key (it’s unclear whether or not an attacker actually downloaded it, but they might have):

HashiCorp was impacted by a security incident with a third party (Codecov) that led to potential disclosure of sensitive information. As a result, the GPG key used for release signing and verification has been rotated.

But it’s not just CodeCov. One pentester discovered hundreds of leaked secrets on public Docker images:

I found a wide variety of secrets, such as 200+ AWS accounts (of which 64 were still alive and 14 of these were root), 1,500+ valid SSH keys, Azure keys, several databases, .npmrc tokens, Docker Hub accounts, PyPI repository keys, many SMTP servers, reCAPTCHA secrets, Twitter API keys, Jira keys, Slack keys, and a few others.

There are multiple ways secrets can get leaked, and it’s unclear which occurred in these case. So let’s go over all of them.

Accidental COPYing

Consider the following common way of copying files into your image:

FROM debian:buster
COPY . /app

By default this will copy everything in the current directory into the image. That everything might include a variety of secrets you didn’t intend to copy in: a .env file with password, for example.

Deleting the file afterwards with RUN rm mysecret won’t help, because of the way Docker layers work. You need to make sure they never get copied in in the first place.

Once the secret is in the Docker image, anyone who can access that image now has access to the secret. This is extra fun when the image is publicly available, but can still hurt you if the image is in a private repository that gets breached.

How can you prevent copying in secrets by mistake?

  • Limited copying: Instead of COPY . /app you might copy only specific files or directories you know you need. For example, COPY setup.py myapp /app.
  • .dockerignore: You can make sure files don’t get COPYed in by adding them to the .dockerignore file.
  • Avoid manually building images: Your development machine is much more likely to have random files lying around than an automated build system, so building public images on your dev machine is more likely to leak files.
  • Store CI secrets as environment variables: If your CI or build environment needs to use secrets, keep them in environment variables rather than files on disk.

Build secrets

Sometimes you need to use a secret as part of building the Docker image, for example the password to your private package repository.

FROM python:3.9

# How do you get the password in?
RUN pip install \
    --extra-index-url https://you@pkgrepo.example.com \
    yourpackage anotherprivatepackage

The obvious ways to get these passwords in, like build arguments and COPY, will result in the secret being embedded in the image.

There are a variety of techniques that are secure, which I cover in a separate article specifically about Docker build secrets.

Runtime secrets

Sometimes you need some secret to actually run your software in production. For example, your web application might require the password to its MySQL database. These are known as runtime secrets, and the important thing to remember is that, once again, you should not store them in your image.

Once they’re in the image, any attacker who gets access to the image can get that secret, e.g. the MySQL password. Plus, it means your image is tied to a specific runtime environment, and it’ll be harder to take your image and run it somewhere else.

Instead, you can pass secrets into your container when it’s run, via a variety of methods:

  • Using environment variables.
  • Bind mounting a volume with the secret.
  • Secret-specific mechanisms like Kubernetes secrets (which actually uses both mechanisms above).
  • In some cloud environments, you can give permissions to containers to talk to the cloud environment, e.g. IAM roles for ECS tasks on AWS.
  • Retrieving them from some sort of external key store.

With all of these mechanisms the secret is never stored in the image itself.

Finding leaks with scanners

Prevention is important, but it will only take you so far. For example, it’s easy to accidentally COPY in a file you didn’t mean to.

So beyond the guidelines given above, it’s also worth using a tool that scans for secrets that accidentally made it through anyway.