“Externally managed environments”: when PEP 668 breaks pip

You’re on a new version of Linux, you try a pip install, and it errors out, talking about “externally managed environments” and “PEP 668”. What’s going on? How do you solve this?

Let’s see:

• What the problem looks like, and what causes it.
• The places you are likely to encounter it.
• A variety of solutions, depending on your use case.

Symptoms: failed installs

Consider the following Dockerfile, using a pre-release of Debian “Bookworm” 12, which will be released in June 2023:

FROM debian:bookworm
RUN apt-get update && apt-get -y install python3 python3-pip
RUN pip install flask

This seems pretty straightforward, similar to many Docker examples you’ll find. And yet when built, it fails:

 > [3/3] RUN pip install flask:
error: externally-managed-environment

× This environment is externally managed
To install Python packages system-wide, try apt install
python3-xyz, where xyz is the package you are trying to
install.

If you wish to install a non-Debian-packaged Python package,
create a virtual environment using python3 -m venv path/to/venv.
Then use path/to/venv/bin/python and path/to/venv/bin/pip. Make
sure you have python3-full installed.

If you wish to install a non-Debian packaged Python application,
it may be easiest to use pipx install xyz, which will manage a
virtual environment for you. Make sure you have pipx installed.

See /usr/share/doc/python3.11/README.venv for more information.

note: If you believe this is a mistake, please contact your Python installation or OS distribution provider. You can override this, at the risk of breaking your Python installation or OS, by passing --break-system-packages.
hint: See PEP 668 for the detailed specification.

Note: Outside any specific best practice being demonstrated, the Dockerfiles in this article are not examples of best practices, since the added complexity would obscure the main point of the article.

Need to ship quickly, and don’t have time to figure out every detail on your own? Read the concise, action-oriented Python on Docker Production Handbook.

Where you’ll encounter this

As we saw above, this problem occurs with the system Python from Debian 12. It’ll also happen in Ubuntu 23.04 or later. This is less relevant for Docker builds, since that’s not a long-term support release, but it will become an issue once Ubuntu 24.04 LTS is released.

And other Linux distributions will likely have this issue as well. For example, as of May 2023, Fedora has a proposal to turn this on.

Older Linux distributions, like Ubuntu 22.04, Debian 11, and current RHEL and clones, do not have this problem.

Additionally, if you use the “official” Python base images provided by Docker, you won’t have any issues:

FROM python:3.11-slim-bullseye
RUN pip install flask

$ docker build .
...
 => => writing image sha256:483c6c3804fe2681acf61d79af326cd81c1d1824cd45f4b7c04bd4f7728  0.0s
$

This image is based on Debian 11, but that’s not why it works. I would expect future python images based off of Debian 12 to build just fine too.

Why the difference? And what is PEP 668 and why does it break things?

System packages vs. pip installs

Linux distributions package a variety of software, both libraries and applications. Some of that software is written in Python.

For example, on Ubuntu 20.04:

• The update-manager package, the GUI for installing system updates, depends on python3-yaml.
• The python3-yaml package is version 5.3.1.
• This is based on the python3-yaml PyPI package, version 6.0 at the time of writing.

So if I start a Python prompt, I can import yaml:

$ docker run -it ubuntu:20.04 bash
root@d67511efab81:/# apt-get update && apt-get install --no-install-recommends python3-pip update-manager
...
root@d67511efab81:/# python3
Python 3.8.10 (default, Mar 13 2023, 10:26:41)
[GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import yaml
>>> yaml.__version__
'5.3.1'

That’s bad if you want your code to run reproducibly: you now have a specific version of Python’s YAML library installed and importable that might not match the one your code expects.

And what if I then pip install --upgrade pyyaml as root?

root@d67511efab81:/# pip install --upgrade pyyaml
...
root@d67511efab81:/# python3
Python 3.8.10 (default, Mar 13 2023, 10:26:41)
[GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import yaml
>>> yaml.__version__
'6.0'

Does the update-manager tool still work with PyYAML 6 instead of the 5.3 it expected? Who knows!

To summarize, we have two core problems:

1. Leakage: System-provided Python packages are unexpectedly importable.
2. Breakage: pip-installed packages can break system packages.

Solving the mismatch

The first problem, that of leakage, is solvable with virtualenvs, which give you isolated environments where any packages installed by the operating system won’t affect you.

root@d67511efab81:/# python3 -m venv myvenv
root@d67511efab81:/# myvenv/bin/python
Python 3.8.10 (default, Mar 13 2023, 10:26:41)
[GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import yaml
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'yaml'

The second problem, of pip install breaking system packages, is solved by PEP 668. PEPs are Python Enhancement Proposals: proposals to improve the Python language or core tooling in some way.

PEP 668 provides a mechanism Linux distributions to tell pip that it can’t install packages outside a virtualenv. And the latest versions of Debian and Ubuntu have decided, quite reasonably, to opt-in to this mechanism. You don’t want a stray pip install breaking your operating system packages!

That’s the source of the “externally managed environment” error: you’re trying to pip install into an environment managed by the Linux distribution.

pip install --user breaks too

Linux distributions that have opted-in to PEP 668 will also prevent pip install --user, a mechanism that lets you install user-specific packages in your home directory. While this mechanism is better than pip install as root, since it definitely won’t overwrite files installed by the operating system, it still has the issue that the modules being imported might be different than the ones system packages expect.

For example, if you pip install --user pyyaml, that might result in update-manager importing a different version of PyYAML than it expected; this is leakage from the user to the system.

Solving the “externally managed environment” problem

There are a number of approaches you can use if you’ve encountered an “externally managed environment” error.

During development: virtualenvs

The most general solution is installing everything in a virtualenv: it’s isolated from the system Python so you can do whatever you want without worrying about leakage or breakage.

$ python -m venv ./myvenv
$ . ./myvenv/bin/activate
(venv)$ pip install flask

For Docker packaging: Don’t use the system Python

First, virtualenvs also work for Docker packaging, so you could just do that.

But if you don’t want to use a virtualenv, there’s another option. The motivation for PEP 668 was preventing conflicts between system packages and user packages. If you’re using a version of Python that was not installed by the operating system, then such conflicts are not an issue.

• python Docker images: For the Docker use case, I expect that future python base images will continue to not have any restrictions on pip install, because the Python they provide is not the operating system Python, it’s a separate install.
• Conda: Similarly, Conda installs its own Python.

For system-packaged applications: system package manager

If you want to install an application written in Python, you can just install the version provided by your Linux distribution, for example with apt or snap on Ubuntu.

For Python tools where you want the latest version: pipx

Consider tools like black or mypy, where the fact they’re implemented in Python is more of an implementation detail. Still, you often want the latest version, and your Linux distribution might be out of date.

One good solution is pipx. pipx lets you install Python applications in virtualenvs, one per application, so they don’t break each other (or system packages!).

Because of the bootstrapping problem (how do you install pipx if pip won’t work?) you’ll want to use the pipx from your Linux distribution.

$ sudo apt-get install -y pipx
...
$ pipx ensurepath

And now you can do:

$ pipx install black
$ pipx install mypy

Packaging is a pain: be kind!

Linux distributions with a focus on stability have different use cases than developers writing code, who in turn have different use cases than developers packaging their code, or developers installing tools. Python sits in the middle, trying to satisfy all these use cases while still allowing your code to run.

While encountering these sort of changes can be frustrating, keep in mind that everyone involved likely spent a lot of time thinking through alternatives and consequences, and didn’t make these changes lightly. I have yet to see any packaging setup that isn’t painful, and that seems unlikely to change in the future.