Staying secure by breaking Docker caching

When building Docker images, caching lets you speed up rebuilding images. But this has a downside: it can keep you from installing security updates from your base Linux distribution. If you cache the image layer that includes the security update… you’re not getting new security updates!

There are a number of ways you can try to balance caching with getting security updates, with different tradeoffs. In this article we’ll cover:

• Caching by default, with recurring rebuild and redeploys.
• Deliberately breaking caching.
• Caching during development, never caching in production builds.

Recap: the problem

I’ve written a separate article about this problem but here’s a recap. Consider the following Dockerfile:

FROM python:3.11-slim
RUN apt-get update && apt-get -y upgrade
COPY requirements.txt .
RUN pip install -r requirements.txt
COPY myapp .
ENTRYPOINT ["python", "-m", "myapp"]

The first time you build it, you will get security updates installed via apt-get. If you’re using caching, when you rebuild the image those apt-get commands won’t get rerun; instead, the cached version will be used.

Time passes, and now you’re months behind on security updates.

Oops.

Note: Outside any specific best practice being demonstrated, the Dockerfiles in this article are not examples of best practices, since the added complexity would obscure the main point of the article.

Need to ship quickly, and don’t have time to figure out every detail on your own? Read the concise, action-oriented Python on Docker Production Handbook.

Solving the problem

When considering alternative solutions, there are multiple goals to balance:

• Timely security updates.
• Fast rebuilds when building the Docker image on developer machines.
• Fast rebuilds when building the production Docker image in CI, as part of the production pipeline.
• Minimizing complexity.

Let’s consider different approaches and how they do on these goals.

Solution #1: Cache by default, automatic recurring uncached rebuilds

This was the solution proposed in the original article on this problem.

1. You don’t change the Dockerfile, so by default apt-get upgrade is cached.
2. You setup an automatic recurring rebuild process in your CI or CD system that rebuilds the image using docker build --pull --no-cache. Using those options means the production image is rebuilt from scratch with no caching whatsoever, ensuring security updates get applied. You can then redeploy this image if necessary. For example, you might do this once a week.

Pros:

• Fast rebuilds during development and CI.
• Security updates happen even if you don’t push new code.

Cons:

• Security updates only happen once a week.
• Somewhat complex.

Solution #2: Invalidate cache whenever code changes

We can change the Dockerfile so that installing security updates is the last step, instead of the first one:

FROM python:3.11-slim
COPY requirements.txt .
RUN pip install -r requirements.txt
COPY myapp .
RUN apt-get update && apt-get -y upgrade
ENTRYPOINT ["python", "-m", "myapp"]

Once a layer is invalidated in the cache, all later layers also have to be rebuilt.

So, with this setup, anytime your code changes, the security updates will also happen. You will still get caching for the pip install and other expensive build commands, however, so this isn’t too bad.

Pros:

• Security updates happen every time you build a new image due to code changes, but you still otherwise have caching for the build.
• Simple.

Cons:

• You only get security updates when you push code; this may be fine if you do frequent deploys.
• Every time you rebuild the image, whether locally or in CI, you have to wait for apt-get update etc., even if there have been no security updates upstream. This is faster than a complete rebuild, though!

Solution #3: Cache in development, never in production

We can split our Dockerfile into a multi-stage build, where the first stage is used by developers, and the second stage is used in production:

FROM python:3.11-slim as no_security_updates
COPY requirements.txt .
RUN pip install -r requirements.txt
COPY myapp .
ENTRYPOINT ["python", "-m", "myapp"]

FROM no_security_updates as production
RUN apt-get update && apt-get -y upgrade

During development, you can do:

$ docker build --target=no_security_updates -t myapp .

To avoid spending time on installing security updates.

In CI, you just build normally (docker build -t myapp . ) and get the final stage as your image.

Pros:

• Security updates happen every time you build a new image due to code changes, but you still otherwise have caching for the build.
• Simple in CI, a bit more complex for developers.

Cons:

• You only get security updates when you push code; this may be fine if you do frequent deploys.
• Every time you rebuild the image in CI you have to wait for apt-get update etc., even if there have been no security updates upstream.

Best practices are situational

The typical advice you’ll get pushes you towards maximizing caching, which means apt-get update and friends always happen first. And depending on your goals, that may be a fine solution!

But sometimes you don’t want caching, in which case what may be a worst practice (no caching most of the time) turns out to be a best practice.