50 years of accumulated technologies, 70+ best practices, and you need to figure it out all on your own

You’re packaging your Python application for production with Docker—and production use means you need to implement best practices: for security, speed, reproducibility, debuggability. And of course you want fast builds and small images.

Unfortunately, to implement those best practices there’s a lot of details you need to get right, because Docker packaging is surprisingly complicated.

Here’s why:

Problem #1: All the technologies

Docker packaging for Python builds on 50 years of accumulated tools, technologies, and techniques:

• Signal handling (a design from the 1970s…),
• Shell scripting,
• the Dockerfile format,
• Docker’s build caching,
• Python packaging (which might involve setup.py, requirements.txt, Poetry, Pipenv, PEP 517, Conda, …),
• Git,
• CI,
• Linux distribution’s packaging,
• Container runtimes and environments,
• and on and on and on.

You have a long todo list already—do you have time to spend learning all the relevant features and misfeatures of these technologies and how they interact?

Problem #2: All the processes

Packaging is also at the intersection of the many organizational processes involved in software development:

• Writing the code,
• Testing the code locally,
• Automated testing and CI,
• Deployments, both new and updates,
• Running in production,
• Debugging production bug reports,
• Security updates,
• Dependency updates,
• and on and on and on.

That means that when packaging your software with Docker, you need to ensure your packaging doesn’t make break these processes, or slow them down, or make them more difficult. Ideally, you’d make these processes work better!

Problem #3: All the details

Those two problems lead to the third problem—you need to get a lot of details right. And you can’t abstract the details away, the details are inherent in the problem domain.

• 50 years of technology accumulation means that Mistakes Were Made and now we get to live with them (✨ Unix signals ✨).
• All of these technologies interact with each other, sometimes in unexpected ways. Docker’s build caching makes your build faster—but it can prevent your system packages from getting security updates. Oops!
• Advice that is good for other programming languages might not work for Python; Alpine Linux as a base image and building your image can literally be 50× slower. You don’t want your team spending all their time waiting for CI to recompile a package for the seventeenth time.
• New software releases can make old advice no longer relevant.

In short, Docker packaging is complicated, complex, and confounding. That’s why I’ve been trying to make it a little simpler.

I did the research—so you don’t have to

Over the past three years I’ve been experimenting, reading, and researching how best to approach Docker packaging for Python applications in production. Not because I particularly enjoy it, but because I don’t want you to have to deal with it. I can do the research once, and then you and everyone else can benefit from what I’ve learned and go do some actually useful work.

A lot of what I learned is written up in the articles you can find elsewhere on this site. The goal is for you to learn in depth, so the articles explain the underlying concepts in detail, and give examples of why alternatives might not work. This comes with some downsides, however:

• As of April 2022, those articles add up to 60,000 words. That takes a while to read!
• They’re organized for learning, not for action.
• They don’t cover all best practices that I know of, just the ones I’ve had time to turn into articles.

And that’s great if you have the time and motivation to read through all those articles, and then find the rest of the details elsewhere on the Internet. But what if you don’t have time, what if you just want to package your application using best practices, right now?

The faster route to Docker packaging

Sometimes you don’t want to learn in-depth; sometimes you just need to get some work done, and quickly. And this is where the Python on Docker Production Handbook comes in. Instead of teaching you all the details like my educational articles do, the Handbook helps you ship your software quickly.

In particular, the Handbook:

• Provides a succinct, practical reference covering over 70 best practices, with examples and links to further reading. At 19,000 words, it’s a third of the length of the free articles elsewhere on this site, even as it covers additional best practices.
• Organizes these best practices into a clear, step-by-step iterative process to packaging your application with Docker, starting with the most important parts like security, and ending with optimizations like build time and image size.
• Keeps you up-to-date with Docker 23.0, BuildKit, Ubuntu 22.04, RHEL 9, Debian 12, and specific information on Poetry, Pipenv, and Conda packaging.

Here’s what readers have to say about the Handbook:

“I’ve found it very helpful, just having a general guide off best practices to anything to do with deployment. I find this an excellent complement to Two Scoops of Django.”

— Joe Selvik

“I’ve been using [it] a lot over the last two weeks and it’s been absolutely fantastic.

Learning about multi-stage builds has meant I can combine a Django/React app into one Dockerfile. This also lead to me discovering compose build sections can target a named stage so now I’ve got local dev using the same Dockerfile, which is great.”

– George Hickman

The included best practices cover the following areas:

• Security.
• Running in CI.
• Easier debugging.
• Operational correctness.
• Reproducible builds.
• Faster builds.
• Smaller images, including multi-stage builds.
• Application-specific best practices.
• Conda.
• Poetry.
• Pipenv.

Start packaging with the Python on Docker Production Handbook (US$79)

Includes the Python on Docker Production Handbook:

• 120-page PDF, plus HTML version for easier copy/pasting of example code.
  
• Focuses specifically on Python running in production.
  
• Provides a step-by-step iterative implementation plan.
  
• Covers over 70 best practices, updated for Docker 23.0, BuildKit, the latest Poetry and Pipenv releases, and more.
  
• All future updates to the book, delivered by email (last updated June 2023–see the changelog).
• 100% money-back guarantee.

Note: I prefer not to take money from teams working on projects for (A) the military, surveillance, or national security (B) prisons or police, (C) fossil fuel extraction or blockchain. If any of these categories apply to you, please don’t purchase this product.

5% of sales in 2024 will donated to UNRWA.

Separately, if you live in a low-income country, email me about purchasing price parity (PPP) discounts.

---

Bundle #1: Save time with a pre-written template
(Pip/Poetry/Pipenv edition, US$359)

Includes:

1. The Python on Docker Production Handbook.
2. The Production-Ready Python Containers template (US$349 value).
3. No questions asked, 100% money-back guarantee.

The Handbook covers a large number of Docker packaging best practices, and tries to explain them as succinctly as possible. But implementing all these best practices still takes time, time you might not have.

You could ship your application even faster if these best practices were already implemented for you. That’s why I’ve created a template that includes a pre-written Dockerfile, build scripts, and CI configuration (GitHub Actions and GitLab CI) implementing the most important and common best practices, from security to speed to small images.

Using this template, you can package your Python application for Docker in as little as an hour. Packaging your second application might only take a few minutes! And you’ll have the benefit of security, speed, and small images from the implemented best practices.

What users are saying:

“The [production-ready] template worked great 🙂

The app using it is now live within the hospital’s ICU unit, and I’ll be using the same template to deploy an API (still in development) on the same infrastructure.”

– Nel Swanepol, University College London

In common cases, and once you know how to use the template, packaging can get very quick indeed. Watch as I package a web application in less than 4 minutes; I recommend opening it full-screen (the button on the bottom-right of the video):

Features:

• Pre-written configuration files (Dockerfile, build scripts, CI configuration).
• Fast builds and small images, including support for multi-stage builds.
• Support for pipenv and poetry, as well as requirements.txt and setup.py.
• Licensed for unlimited use inside your organization (see the license for details).
• Save 20% of the template’s normal price.
• 100% money-back guarantee.

Note: I prefer not to take money from teams working on projects for (A) the military, surveillance, or national security (B) prisons or police, (C) fossil fuel extraction or blockchain. If any of these categories apply to you, please don’t purchase this product.

5% of sales in 2024 will donated to UNRWA.

Separately, if you live in a low-income country, email me about purchasing price parity (PPP) discounts.

---

Bundle #2: Save time with a pre-written template
(Conda edition, US$359)

Includes:

1. The Python on Docker Production Handbook.
2. The Production-Ready Conda Containers template (US$349 value).
3. No questions asked, 100% money-back guarantee.

Want to get started even faster? Instead of having to implement the Docker packaging best practices in the handbook yourself, the production-ready template implements many of them for you, in this case with a focus on Conda-based applications:

• Includes pre-written Dockerfile, build scripts, and CI configuration (GitHub Actions and GitLab CI).
• Specifically designed for Conda-based applications.
• Massively smaller images (hundreds of megabytes less!).
• Faster builds with Mamba and layer caching.
• Licensed for unlimited use inside your organization (see the license for details).
• Save 20% of the template’s normal price.
• 100% money-back guarantee.

Note: I prefer not to take money from teams working on projects for (A) the military, surveillance, or national security (B) prisons or police, (C) fossil fuel extraction or blockchain. If any of these categories apply to you, please don’t purchase this product.

5% of sales in 2024 will donated to UNRWA.

Separately, if you live in a low-income country, email me about purchasing price parity (PPP) discounts.

Still not sure?

Read my article on how to decide whether to buy a product, and how to convince your boss to pay for it. And here’s a few more reader reviews:

“We applied the secrets handling and multi-stage builds from the Handbook. Having said that, everything in it was an eye opener.”

— Gregory Ronin, Software Enginer at Deloitte Digital

“It’s super valuable. I now have a big list of TODOs to update our build scripts and Dockerfiles.”

– Eric Pederson

Python on Docker Production Handbook	Bundle #1: + Pip/Poetry/Pipenv Template	Bundle #2: + Conda Template
Just the handbook: 120-page PDF, plus HTML version. Covers 70+ best practices. All updates to 1st edition. 100% money-back guarantee. Learn more	The handbook, plus: Ship faster with a template. Best practices implemented for you. Supports Pip, Poetry, Pipenv. Faster builds, smaller images. Learn more	The handbook, plus: Ship faster with a template. Best practices implemented for you. Specifically designed for Conda. Faster builds, smaller images. Learn more
Buy now US$79	Buy now US$359	Buy now US$359

Note: I prefer not to take money from teams working on projects for (A) the military, surveillance, or national security (B) prisons or police, (C) fossil fuel extraction or blockchain. If any of these categories apply to you, please don’t purchase this product.

5% of sales in 2024 will donated to UNRWA.

Separately, if you live in a low-income country, email me about purchasing price parity (PPP) discounts.

Changelog

As mentioned above, all of the purchase options will includes updates to the 1st edition of the Handbook; here are the updates that have been made so far.

June 19, 2023

Updates:

• Updated with the release of Debian “Bookworm” 12.
• BuildKit is now stable and on by default on all platforms as of Docker v23.0 and later.
• Jake command-line has changed.
• Updated for newer Poetry versions.
• Other small updates.

More best practices:

• Understanding system-managed package installation errors.

September 19, 2022

Updates:

• Newer Trivy fixes remaining commercial usage issues; it also adds support for Alma Linux and RockyLinux.
• conda-lock supports pip packages too in its released version, and has a new file format.
• Podman supports build secrets and mount caches.
• Poetry 1.2 update: new installer, plus poetry-plugin-export is now separate.
• Support for Pipenv 2022.4.8 and later: new pipenv requirements command.
• Ubuntu 22.04, RHEL 9.
• Latest BuildKit backend is now v1.4.
• Switched examples to Python 3.10 instead of 3.9.
• PEP 656 means Alpine-compatible wheels are starting to become available.
• Compose v2 has initial support for build secrets.

More best practices:

• Scanning for leaked secrets.
• Dealing with new ARM (“Silicon”/M1/M2) Macs.

January 26, 2022

• New best practices:
  • Security scans for Conda with Jake.
  • CentOS alternatives.
• Fix conda activate in the face of scripts with semi-broken activation scripts, by disabling bash strict mode.
• Update with some changes to trivy security scanner.
• Removed the safety security scanner, since the non-commercial usage restriction is too limiting.
• Note that conda-lock will support pip packages in the near future.

August 31, 2021

• Debian “Bullseye” 11 has replaced Buster as the stable Debian of choice.
• Latest BuildKit backend is now v1.3.
• Documented tini -g.
• Added note on dealing with pip packages when using conda-lock.
• New best practices:
  • Conda environments require activation.
  • Faster Conda installs with Mamba.

June 1, 2021

• Added best practices:
  • Don’t leak secret files.
  • Don’t leak runtime secrets.
  • Don’t store temporary files in layers.
  • Get rid of unneeded files.
• Added --nodocs to dnf instructions, for even smaller images.
• Added an alternative method for activating Conda environments.

March 16, 2021

• When using BuildKit you should always set --build-arg BUILDKIT_INLINE_CACHE=1 if you want --cache-from to work.
• Noted that official Python Docker image is slower than Ubuntu.

February 8, 2021

• The Quickstart has been renamed to the Handbook; at 100 pages, it’s getting more than just introductory.
• New best practices:
  • Installing packages with pipenv without exporting.
  • Keeping pipenv separate from application code.
  • Recommend docker build --label over LABEL.
  • Caching package downloads across builds using BuildKit.
  • Avoiding dev dependency installs in Poetry.
  • Conda dependency locking using conda-lock.
• Removed references to CentOS, as it is no longer a stable base image.
• Added link to RedHat’s Docker base image.
• Noted Podman can be used with BuildKit.
• Noted need for --keep-outdated when using pipenv lock.
• Documented how to make Docker Compose use BuildKit.
• Noted need to pass --no-capture-output to conda run (thanks to Joe Selvik).

December 17, 2020

Updates for Docker 20.10 and a stable BuildKit.

• Added new chapter covering different Docker releases and BuildKit.
• Switched all BuildKit examples to use the new stable docker/dockerfile:1.2 version.
• Documented getting BuildKit secrets from environment variables.

November 13, 2020

• Documented PEP 517 Poetry installation usage.
• A large number of minor code fixes throughout, as well as some typos in the text.
• HTML version is now included, for easier copy/pasting.

October 22, 2020

• Added three best practices for Poetry.
• Added best practice on adding .git to .dockerignore, and what do when you can’t.

October 14, 2020

Added three best practices for smaller Conda-based images.

June 17, 2020

More best practices:

• Various BuildKit features that help speed up builds.
• Dropping capabilities.
• Avoiding listening on ports < 1024.
• Running a different command altogether based on command-line arguments.
• Bytecode compilation.
• Additional image size optimization in multi-stage builds.
• Warm up the build cache for per-branch builds.
• Requirements for running on Heroku and Google Cloud Run.
• BuildKit ssh-agent forwarding.
• BuildKit secrets when using Docker Compose.

Other tweaks and improvements throughout the text.

June 8, 2020

The Checklist has been renamed, and is now known as a Quickstart. To help make that change:

• Added a new introductory chapter with a plan to help you figure out which best practices to implement when.
• Tweaked the chapter structure.

Additionally:

• Added a best practice on finding large layers.
• Split off init into its own best practice.
• Explained the goal of responding to health checks quickly more broadly, rather than in specific implementation terms of process/thread pool.
• Added more nuance to the section on updating dependencies once a month.
• Restored the Pipenv instructions, since it’s now being maintained again.
• Make the DEBIAN_FRONTEND best practice standalone.

June 2, 2020

Added multiple new best practices:

• Additional tips on improving build caching.
• A better default tag.
• For better reproducibility, you can create a custom base image.
• Size checks for images.
• Security scanners.

Also updated existing best practices:

• You can make a build arg available at runtime by using ENV.
• For build secrets another alternative is short term keys.

April 27, 2020

• Added new best practice on timely security updates via automatic notifications.
• Switched some examples from shell session transcripts to Dockerfile or shell script.
• Noted docker build support for targeting named stages earlier in the checklist.

April 1, 2020

• Documented two-stage install with Poetry 1.0+.
• Added link to docker-autoheal.

February 24, 2020

Added many more examples:

• Configuring logging.
• A smoke test.
• Passing in secrets with BuildKit.
• .dockerignore file.
• System package upgrade script for CentOS/RHEL.
• Dockerfile healthcheck.
• And a few more expanded examples here and there.