“Externally managed environments”: when PEP 668 breaks pip
You’re on a new version of Linux, you try a pip install
, and it errors out, talking about “externally managed environments” and “PEP 668”.
What’s going on?
How do you solve this?
Let’s see:
- What the problem looks like, and what causes it.
- The places you are likely to encounter it.
- A variety of solutions, depending on your use case.
Symptoms: failed installs
Consider the following Dockerfile, using a pre-release of Debian “Bookworm” 12, which will be released in June 2023:
FROM debian:bookworm
RUN apt-get update && apt-get -y install python3 python3-pip
RUN pip install flask
This seems pretty straightforward, similar to many Docker examples you’ll find. And yet when built, it fails:
> [3/3] RUN pip install flask:
error: externally-managed-environment
× This environment is externally managed
To install Python packages system-wide, try apt install
python3-xyz, where xyz is the package you are trying to
install.
If you wish to install a non-Debian-packaged Python package,
create a virtual environment using python3 -m venv path/to/venv.
Then use path/to/venv/bin/python and path/to/venv/bin/pip. Make
sure you have python3-full installed.
If you wish to install a non-Debian packaged Python application,
it may be easiest to use pipx install xyz, which will manage a
virtual environment for you. Make sure you have pipx installed.
See /usr/share/doc/python3.11/README.venv for more information.
note: If you believe this is a mistake, please contact your Python installation or OS distribution provider. You can override this, at the risk of breaking your Python installation or OS, by passing --break-system-packages.
hint: See PEP 668 for the detailed specification.
Note: Outside any specific best practice being demonstrated, the Dockerfiles in this article are not examples of best practices, since the added complexity would obscure the main point of the article.
Need to ship quickly, and don’t have time to figure out every detail on your own? Read the concise, action-oriented Python on Docker Production Handbook.
Where you’ll encounter this
As we saw above, this problem occurs with the system Python from Debian 12. It’ll also happen in Ubuntu 23.04 or later. This is less relevant for Docker builds, since that’s not a long-term support release, but it will become an issue once Ubuntu 24.04 LTS is released.
And other Linux distributions will likely have this issue as well. For example, as of May 2023, Fedora has a proposal to turn this on.
Older Linux distributions, like Ubuntu 22.04, Debian 11, and current RHEL and clones, do not have this problem.
Additionally, if you use the “official” Python base images provided by Docker, you won’t have any issues:
FROM python:3.11-slim-bullseye
RUN pip install flask
$ docker build .
...
=> => writing image sha256:483c6c3804fe2681acf61d79af326cd81c1d1824cd45f4b7c04bd4f7728 0.0s
$
This image is based on Debian 11, but that’s not why it works.
I would expect future python
images based off of Debian 12 to build just fine too.
Base image | Plain RUN pip install works? |
---|---|
ubuntu:23.04 |
No |
debian:12 (Bookworm) |
No |
python:3.11-slim-bullseye |
Yes |
python:3.11-slim-bookworm |
Not available yet, but likely yes |
Why the difference? And what is PEP 668 and why does it break things?
System packages vs. pip
installs
Linux distributions package a variety of software, both libraries and applications. Some of that software is written in Python.
For example, on Ubuntu 20.04:
- The
update-manager
package, the GUI for installing system updates, depends onpython3-yaml
. - The
python3-yaml
package is version 5.3.1. - This is based on the
python3-yaml
PyPI package, version 6.0 at the time of writing.
So if I start a Python prompt, I can import yaml
:
$ docker run -it ubuntu:20.04 bash
root@d67511efab81:/# apt-get update && apt-get install --no-install-recommends python3-pip update-manager
...
root@d67511efab81:/# python3
Python 3.8.10 (default, Mar 13 2023, 10:26:41)
[GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import yaml
>>> yaml.__version__
'5.3.1'
That’s bad if you want your code to run reproducibly: you now have a specific version of Python’s YAML library installed and importable that might not match the one your code expects.
And what if I then pip install --upgrade pyyaml
as root?
root@d67511efab81:/# pip install --upgrade pyyaml
...
root@d67511efab81:/# python3
Python 3.8.10 (default, Mar 13 2023, 10:26:41)
[GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import yaml
>>> yaml.__version__
'6.0'
Does the update-manager
tool still work with PyYAML 6 instead of the 5.3 it expected? Who knows!
To summarize, we have two core problems:
- Leakage: System-provided Python packages are unexpectedly importable.
- Breakage:
pip
-installed packages can break system packages.
Solving the mismatch
The first problem, that of leakage, is solvable with virtualenvs, which give you isolated environments where any packages installed by the operating system won’t affect you.
root@d67511efab81:/# python3 -m venv myvenv
root@d67511efab81:/# myvenv/bin/python
Python 3.8.10 (default, Mar 13 2023, 10:26:41)
[GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import yaml
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'yaml'
The second problem, of pip install
breaking system packages, is solved by PEP 668.
PEPs are Python Enhancement Proposals: proposals to improve the Python language or core tooling in some way.
PEP 668 provides a mechanism Linux distributions to tell pip
that it can’t install packages outside a virtualenv.
And the latest versions of Debian and Ubuntu have decided, quite reasonably, to opt-in to this mechanism.
You don’t want a stray pip install
breaking your operating system packages!
That’s the source of the “externally managed environment” error: you’re trying to pip install
into an environment managed by the Linux distribution.
pip install --user
breaks too
Linux distributions that have opted-in to PEP 668 will also prevent pip install --user
, a mechanism that lets you install user-specific packages in your home directory.
While this mechanism is better than pip install
as root, since it definitely won’t overwrite files installed by the operating system, it still has the issue that the modules being imported might be different than the ones system packages expect.
For example, if you pip install --user pyyaml
, that might result in update-manager
importing a different version of PyYAML than it expected; this is leakage from the user to the system.
Solving the “externally managed environment” problem
There are a number of approaches you can use if you’ve encountered an “externally managed environment” error.
During development: virtualenvs
The most general solution is installing everything in a virtualenv: it’s isolated from the system Python so you can do whatever you want without worrying about leakage or breakage.
$ python -m venv ./myvenv
$ . ./myvenv/bin/activate
(venv)$ pip install flask
For Docker packaging: Don’t use the system Python
First, virtualenvs also work for Docker packaging, so you could just do that.
But if you don’t want to use a virtualenv, there’s another option. The motivation for PEP 668 was preventing conflicts between system packages and user packages. If you’re using a version of Python that was not installed by the operating system, then such conflicts are not an issue.
python
Docker images: For the Docker use case, I expect that futurepython
base images will continue to not have any restrictions onpip install
, because the Python they provide is not the operating system Python, it’s a separate install.- Conda: Similarly, Conda installs its own Python.
For system-packaged applications: system package manager
If you want to install an application written in Python, you can just install the version provided by your Linux distribution, for example with apt
or snap
on Ubuntu.
For Python tools where you want the latest version: pipx
Consider tools like black
or mypy
, where the fact they’re implemented in Python is more of an implementation detail.
Still, you often want the latest version, and your Linux distribution might be out of date.
One good solution is pipx
.
pipx
lets you install Python applications in virtualenvs, one per application, so they don’t break each other (or system packages!).
Because of the bootstrapping problem (how do you install pipx
if pip
won’t work?) you’ll want to use the pipx
from your Linux distribution.
$ sudo apt-get install -y pipx
...
$ pipx ensurepath
And now you can do:
$ pipx install black
$ pipx install mypy
Packaging is a pain: be kind!
Linux distributions with a focus on stability have different use cases than developers writing code, who in turn have different use cases than developers packaging their code, or developers installing tools. Python sits in the middle, trying to satisfy all these use cases while still allowing your code to run.
While encountering these sort of changes can be frustrating, keep in mind that everyone involved likely spent a lot of time thinking through alternatives and consequences, and didn’t make these changes lightly. I have yet to see any packaging setup that isn’t painful, and that seems unlikely to change in the future.