When should you upgrade to Python 3.11?

Python 3.11 is out now–but should you switch to it immediately? And if you shouldn’t upgrade just yet, when should you?

Immediately after the release, you probably didn’t want to upgrade just yet. Now that it’s been out for longer, but you might still have issues, but upgrading is worth trying. To understand why, we need to consider Python packaging, the software development process, and take a look at the history of past releases.

The problems with a new major Python release

As with many open source projects, Python, Python libraries, and most of the toolchain and packages mentioned in this article are based on volunteer labor. None of this should be read as a complaint towards the people doing the maintenance, they’re doing hugely valuable work for free, and everything takes time. In addition, because upgrades involve so many different groups, coordination and releases take even more time.

With that in mind, let’s consider the problems with using 3.11 immediately after a release, and in the following months:

1. Missing binary packages

To begin with, not all packages can be installed on 3.11 yet. For example, psycopg2-binary provides bindings to the PostgreSQL database; it had tens of million of downloads from PyPI over the last month. When I tried to install it on Python 3.11 on the day of 3.11’s release, things don’t go well:

$ python3.11 -m pip install psycopg2-binary
...
pg_config is required to build psycopg2 from source.  Please add the directory
containing pg_config to the $PATH or specify the full executable path with the
option:
...

What’s going on?

Typically, Python package maintainers upload compiled versions of their packages–known as “wheels”–to PyPI. When you install a package, you can just download the binary wheel and don’t need to compile it (unless you’re using Alpine Linux, in which case wheels may or may not be available).

But at least for a short time after 3.11’s release, some packages won’t have wheels for Python 3.11 uploaded yet. In this case, psycopg2-binary added wheels for 3.11 on Oct. 25, the very next day.

In general as time goes on package maintainers are putting in more and more effort to shorten this window. Consider NumPy, for example:

• For the 3.9 release, it did not have wheels available until a while after the release.
• For 3.10, it had a Linux wheel available the day of 3.10’s release, but not macOS and Windows, which took a bit longer.
• For 3.11, wheels were available for all platforms even before 3.11 was released.

Personally I had branches ready for some of my own projects to build 3.11 wheels, but didn’t do any releases in advance because GitHub Actions’ release candidate Python versions are just different to sometimes make life difficult. Some projects required writing code to support changed APIs in Python internals, but will eventually get released—we’ll cover that next.

Important: Make sure you upgrade pip before using it. Older versions might not support the binary wheels for 3.11 on Linux, and instead pip will try to build from source.

2. Incompatible packages

Sometimes supporting a new release is just a matter of recompiling the code. That means you can can compile it yourself, with enough work, and if not the maintainer will usually release a new wheel pretty quickly. In other cases, however, the code is incompatible and requires some work to support new Python releases.

The Python 3.11 support for the Numba project, for example, is still a work-in-progress as of Dec 8, 2022. Apparently it took them 6 months post-release until they had Python 3.9 support, and 3 months after 3.10.

For my own projects, some should just work, but e.g. porting the Sciagraph performance and memory profiler took a couple of months because of some of the major changes to Python’s internals.

Even if the code is compatible, if dependencies aren’t compatible that can still delay a release. For example, when I started porting the Fil to 3.10, I couldn’t do a release because the test suite relied on the numexpr package, and numexpr wouldn’t release 3.10 wheels until NumPy had complete 3.10 wheels, which didn’t happen immediately. So there was a tree of dependencies blocking a new release.

Given many Python packages are maintained by volunteers with limited free time and working independently, it just takes time for updates to percolate through the system.

3. Conda support

If you’re using Conda-Forge, the Python 3.11 rollout happens pretty quickly. A lot of it is automated, one of the benefits of Conda-Forge over PyPI, so as long as the underlying packages already support 3.11, it should happen pretty quickly now that it’s happening.

5. Bugs in Python

Python 3.10.0 was released in October 2021. 3.10.1 was released 2 months later, with a long list of bugfixes.

Of course, there’s always another bug fix release in the future, but given a 12-month window between major releases, lots of new code will get written but not extensively used. Which means a follow-up release with plenty of bug fixes. Python 3.11.1 was released in December 2022, with many bug fixes as libraries and then applications start switching and encountering more issues.

6. Lack of toolchain support

New versions of Python often have new syntax, and that is the case with Python 3.11: you can catch multiple exceptions at once with except*. However, other tools need to support the new syntax too—autoformatters, linters, and so on. Until they do, you can’t use the new syntax. For example, as of December 2022 the PyLint linter still considers this syntax to be an error.

That being said, Python 3.11 has significant benefits even if you ignore the new syntax; it’s much faster, for example. So you can just choose not to use the new syntax in the short term.

3.11 involved more significant internal changes than usual, so it also impacted other tools even if they do run. For example, coverage.py is much slower. It still runs, though, and having production code run much faster seems like a reasonable tradeoff.

Is upgrading worth it?

Given that it takes work to upgrade—some additional testing, some tweaks to your code—it can be tempting to put off upgrading Python versions indefinitely. Why worry about incompatibilities, new versions, and what not, when you can just stick with your current version indefinitely?

The problem is that Python isn’t supported indefinitely, nor do libraries support all Python versions indefinitely; eventually you’ll stop getting security updates and bug fixes. Python 3.6 ended security updates in December 2021, and Python 3.7 will end security updates in June 2023.

So sooner or later you will have to upgrade. And if you’re running on a 5-year-old version of Python, switching becomes a Big Deal—you’ll often end up dealing with more significant cross-version changes in both Python and in libraries at the same time. Which makes upgrading scary.

Instead of one massive scary upgrade event every few years, it’s much safer to have a continuous, ongoing process of smaller upgrades. Whenever a new major Python version comes out, or a new major library version, wait a bit, and then switch.

The week Python 3.11 was released, you probably didn’t want to upgrade, but you certainly wanted to be using 3.10. If you’re far enough behind, do a series of upgrades: 3.8, 3.9, 3.10, and eventually, 3.11.

When should you switch to Python 3.11?

At a minimum, after a major new release you will need to wait until:

1. All your libraries explicitly support the new Python release.
2. All the tools you rely on explicitly support the new Python release.

Now that some time has passed from the initial release, and 3.11.1 has been released, it’s probably worth trying it out. If dependencies you require are still missing 3.11 support, keep trying again every month.