Why new Macs break your Docker build, and how to fix it

by
Last updated 03 May 2024, originally created 20 Jun 2022

One of the promises of Docker is reproducibility: you can build an image on a different machine, and assuming you’ve done the appropriate setup, get the same result. So it can be a little confusing when you try to build your Python-based Dockerfile on a new Mac, and everything starts failing. What used to work before—on an older Mac, or on a Linux machine—fails in completely unexpected ways.

The problem is that the promise of reproducibility relies on certain invariants that don’t apply on newer Macs. The symptoms can be non-obvious, though, so in this article we’ll cover:

  • Common symptoms of the problem when using Python.
  • The cause of the problem: a different CPU instruction set.
  • Solving the problem by ensuring the code is installable or compilable.
  • Solving problem with CPU emulation, some of the downsides of this solution, and future improvements to look forward to.
  • A takeaway for maintainers of open source Python packages.

Identifying the problem

Symptom #1: You need a compiler

Consider the following Dockerfile:

FROM python:3.9-slim
RUN pip install murmurhash==1.0.6

If we build it on a Linux desktop, all is well:

linux:$ docker build .
...
Successfully built bda44f5fa6cc

On a Mac Mini M1, however, things go wrong:

macos:% docker build .
...
> [2/2] RUN pip install murmurhash==1.0.6:
Collecting murmurhash==1.0.6
Downloading murmurhash-1.0.6.tar.gz (12 kB)
...
creating build/temp.linux-aarch64-cpython-39/murmurhash
gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/usr/local/include/python3.9 -I/tmp/pip-install-qitioo67/murmurhash_05dd4f37b6414216a03a1a2d0e285374/murmurhash/include -I/usr/local/include/python3.9 -c murmurhash/MurmurHash2.cpp -o build/temp.linux-aarch64-cpython-39/murmurhash/MurmurHash2.o -O3 -Wno-strict-prototypes -Wno-unused-function
error: command 'gcc' failed: No such file or directory
...

Why do we need a compiler on the Mac, but not on Linux?

Symptom #2: Missing packages

Let’s try a different Dockerfile:

FROM python:3.9-slim
RUN pip install filprofiler==2022.05.0

Again, everything works on fine on my Linux machine:

linux:$ docker build .
...
Successfully built 1d8aa3de92bb

But on the Mac, it fails with a new error:

% docker build .
...
ERROR: Could not find a version that satisfies the requirement filprofiler==2022.05.0 (from versions: none)
ERROR: No matching distribution found for filprofiler==2022.05.0
...

The problem: different hardware and wheel availability

So why do the Linux machine and Mac have such different outcomes? It’s not the operating system; it’s the result of a different CPU instruction set combined with lack of binary wheels.

CPU instruction sets

The CPU instruction set is the language the CPU speaks, the set of binary instructions it can interpret. My Linux computer is running an Intel chip, which uses the x86_64 instruction set, also known as AMD64 since it was first created by AMD. CPUs from Intel and AMD uses this instruction set.

Older Macs use this instruction set as well, but new Macs with M1 or M2 processors use the ARM64 instruction set, aka aarch64; Apple calls these “Apple Silicon” CPUs just to throw in a little meaningless terminology confusion. ARM64 and x86_64 instruction sets are different languages; a CPU that speaks one can’t understand the other.

That means binary code like an executable or Python extension compiled for x86_64 can’t run on a ARM64 CPU, and vice versa.

Note: AMD64 and ARM64 are visually similar, but different instruction sets, so be careful when reading those names.

How Docker deals with CPU instruction sets (the simple version)

Since executable code needs to match the CPU, you need different Docker images for different CPU instruction sets. Even though on both machines the base image was python:3.9-slim, in practice different images were chosen.

If we look at the images for the python:3.9-slim tag, we can see there are actually multiple images available, covering multiple different architectures. The relevant ones for our purposes are the linux/amd64 image, used on Intel or AMD CPUs, and the linux/arm64/v8 used on newer Macs. Depending what CPU you’re using, Docker will pull the respective image.

Why is there linux in both cases? Aren’t we on macOS? In order to meet its build-once-run-everywhere promise, Docker typically runs on Linux. Since macOS is not Linux, on macOS this is done by running a virtual machine in the background, and then the Docker images run inside the virtual machine. So whether you’re on a Mac, Linux, or Windows, typically you’ll be running linux Docker images.

In short: on an Intel/AMD PC or Mac, docker pull will pull the linux/amd64 image. On a newer Mac using M1/M2/Silicon chips, docker pull will the pull the linux/arm64/v8 image.

Note: Docker also supports Windows-oriented images, but that’s out of scope for this article so I won’t mention it again.

Python, Docker and CPU instruction sets

How does Python usage interact with Docker images and CPU instrunction sets? If we’re using pure Python code, it’s almost invisible.

  • On Intel or AMD CPUs, python:3.9-slim is the x86_64 aka amd64 image, which has a Python executable compiled for x86_64.
  • On ARM64 machines like my Mac Mini, python:3.9-slim is the ARM64 image, which has a Python executable compiled for ARM64.

In either case, pure Python will Just Work, because it’s interpreted at runtime: there’s no CPU-specific machine code, it’s just text that the Python interpreter knows how to run.

The problems start when we start using compiled Python extensions. These are machine code, and therefore you need a version that is specific to your particular CPU instruction set.

The impact of Python wheel and source availability

In order to make it easier to install compiled Python extensions, package maintainers can upload pre-compiled “wheels” to PyPI. As you might expect, wheels are typically specific to a particular CPU instruction set. And the wheels we care about for Docker packaging will always be the Linux wheels, since Docker images we’re focusing on are always Linux-based.

We can now explain the two sets of symptoms we saw earlier.

When you pip install filprofiler==2022.05.0 on a M1/M2 Mac inside Docker, pip will look at the available files for that version, and then:

  • Try to download a manylinux aarch64 wheel, since using Docker means it’s actually running on a Linux virtualmachine.
  • Since that version of the Fil memory profiler doesn’t have any such wheel uploaded, pip will then look for a source .tar.gz file.
  • Since it can find that either (I’ve never gotten around to making one), installation will fail with a “no matching distribution” error message.

If you were running on a x86_64 machine—an older Mac, or a PC—it would instead be looking instead for amd64 wheels, which do exist.

When you pip install murmurhash==1.0.6 on a M1/M2 Mac inside Docker, again it looks at the available files:

  • It wants to download a manylinux aarch64 wheel.
  • Since none exist, it will download the source tarball.
  • It unpacks it, and tries to compile the package from source.
  • At this point it fails, because there’s no gcc compiler installed in the build image I was using.

In short, a lack of manylinux aarch64 wheels for these packages explains both the errors we were seeing. Importantly, macos arm64 wheels are not sufficient; since we’re running inside Docker, we need Linux wheels.

Solution #1: Get the code to install

So how we can get the code installed if there aren’t wheels?

First, it’s possible the relevant wheels are available in newer versions of the libraries. So try upgrading, and see if that helps. If there is no aarch64 wheel, politely file a bug with the upstream maintainer, and with any luck they will—eventually, when and if they have time—build these wheels for future release.

Second, if that doesn’t work, you can just make sure the source code packages compile. If you have a compiler installed in your Docker image and any required native libraries and development headers, you can compile a native package from the source code. Basically, you add a RUN apt-get upgrade && apt-get install -y gcc and iterate until the package compiles successfully. The problem is that image builds will get slower (solvable with caching), and the images will be larger (solvable with multi-stage builds).

Third, you can pre-compile wheels, store them somewhere, and install those directly instead of downloading the packages from PyPI.

Solution #2: Run x86_64 Docker images instead

The other option is to run x86_64 Docker images on your ARM64 Mac machine, using emulation. Docker is packaged with software that will translate or emulate x86_64 machine code into ARM64 machine code on the fly. There are two versions:

  • Using Rosetta, which is a built-in extra-fast macOS emulation system. It has some limitations, however, e.g. not supporting AVX instructions. This is on by default in Docker Desktop as of October 2023, assuming you’re running macOS 14.1 or later; if you’re on macOS 13 you can manually enable it in settings.
  • A much slower emulator that might support more software, which will be used if you disble Rosetta.

You can enable this on the command-line:

macos:% docker build --platform linux/amd64 -t myimage .
...
macos:% docker run --platform linux/amd64 myimage

Or with an environment variable:

macos:% export DOCKER_DEFAULT_PLATFORM=linux/amd64
macos:% docker build -t myimage .
...
macos:% docker run myimage

Or you can set this in the Dockerfile:

FROM --platform=linux/amd64 python:3.9-slim
RUN pip install murmurhash==1.0.6

In Compose you can also use the platform option per-service.

With any of these options you will end up installing manylinux amd64 wheels, which are what most Python packages will provide at the moment. And that means you’ll be able to build your image. And thanks to Rosetta, it won’t be too slow.

If you go with this approach, which of the configuration options above should you use? In general, the environment variable is too heavy-handed and should be avoided, since it will impact all images you build or run. Given the speed impact, you don’t for example want to run your postgres image with emulation, to no benefit. You’re much better off specifying on a per-image basis whether it’s emulated or not.

Solution #3: Switch to Conda-Forge

The Conda-Forge package repository operates on a completely different model than PyPI, where the people who build the packages need not be the authors of the package. The maintainers are aided by centralized, semi-automated upgrades. As a result, the addition of ARM support (for both Mac and Linux packages) is less of a burden on individual maintainers, and in general you can expect a higher percentage of packages to be available for ARM. The caveat, of course, is that Conda-Forge has fewer Python packages than PyPI.

You can learn more about the differences between pip and Conda.

A side note for Python package maintainers

For maintainers of Python open source packages, the new Macs have added extra work: providing macOS wheels compiled for ARM64. Both Fil (which I maintain) and murmurhash provide these wheels in their latest version. However, as we’ve seen, macOS ARM64 wheels are not sufficient to fully support ARM64 Mac users.

If an ARM64 Mac user runs Docker, they will also want manylinux aarch64 wheels… and both Fil and murmurhash lacked those (as of June 2022 when I originally wrote this article). This isn’t an accusation or complaint, since I maintain Fil I’m as guilty of this as anyone else (and I maintain other packages with the same issue.) Since ARM64-based build machines are still not readily available everywhere it’s annoying to setup.

Nonetheless, best practices as this point should be to provide both macOS and Linux ARM64 wheels. If you’re using cibuildwheel to build wheels, there’s some documentation of how to do this. If you’re packaging Rust code with Maturin, the GitHub Action messense/maturin-action can also do cross-compilation.

Beyond Mac users, manylinux aarch64 wheels are also helpful for users who are switching to ARM64-based Linux machines in the cloud. AWS, for example, claims its Graviton 3 machines have “up to 40% better price performance over comparable current generation x86-based instances.”

Takeaways

Now that Rosetta is available by default on Docker Desktop for macOS, the situation should be better. Some images won’t work, however, at which point you may be stuck with super-slow emulation.