When you should switch to Python 3.10
Python 3.10 is now available–but should you switch to it immediately? And if not now, when?
The short answer after its immediate release was, no, you probably don’t want to switch immediately; quite possibly you can’t switch immediately. Now that some time has passed, we’re getting closer to yes, or at least a maybe. To understand why, we need to consider Python packaging, the software development process, and take a look at the history of past releases.
We can then make a guess about when Python 3.10 will actually be usable.
The problems with a new major Python release
I originally wrote this article on October 5th, 2021, the day after 3.10 was released… and far too early to start using Python 3.10.
As with many open source projects, Python, Python libraries, and most of the toolchain and packages mentioned in this article are based on volunteer labor. None of this should be read as a complaint towards the people doing the maintenance, they’re doing hugely valuable work for free, and everything takes time. In addition, because upgrades involve so many different groups, coordination and releases take even more time.
With that in mind, let’s consider the problems with using 3.10 on its release day, and in the following months:
1. Missing Docker images
Update Oct 12, 2021: The final Docker 3.10 image is now available.
Let’s try running Python 3.10 under Docker:
$ docker run -it python:3.10 Unable to find image 'python:3.10' locally docker: Error response from daemon: manifest for python:3.10 not found: manifest unknown: manifest unknown.
As it turns out, there is no Python 3.10 Docker image for the final release.
If you want to use Python 3.10, you can use a release candidate, e.g.
No doubt it will be available within the next few days, maybe even today, but early on even installing a new release can be difficult. And that’s just our first problem.
2. Missing binary packages
Update Nov 11, 2021:
psycopg2-binaryadded 3.10 support.
Update Nov 16, 2021:
matplotlibadded 3.10 support.
Another problem is that not all packages support 3.10 yet.
psycopg2-binary provides bindings to the PostgreSQL database; it had 17 million downloads from PyPI over the last month.
When I try to install it on Python 3.10 things don’t go well:
$ docker run python:3.10-rc-slim pip install psycopg2-binary ... Error: pg_config executable not found. pg_config is required to build psycopg2 from source. ...
What’s going on?
Typically, Python package maintainers upload compiled versions of their packages–known as “wheels”–to PyPI. When you install a package, you can just download the binary wheel and don’t need to compile it (unless you’re using Alpine Linux).
But it’s so soon after 3.10’s release, many packages don’t have wheels for Python 3.10 yet.
In this case,
psycopg2-binary still doesn’t have wheels for 3.10.
To be fair, things seem to be improving on this front: for the 3.9 release, NumPy and Pandas did not have wheels available this soon after the release. This time around, they do. That being said, the NumPy package is limited to Linux, and mostly exists for testing purposes. As of Oct 12, 2021, Windows and macOS are still not supported. And since Pandas depends on NumPy, it won’t work on Windows or macOS either until the NumPy package is released.
There are plenty of other projects missing 3.10 wheels; to pick another random example,
matplotlib (with 27M downloads a month from PyPI) is also missing a 3.10 binary wheel.
Important: Make sure you upgrade pip, otherwise you might not get the latest binary wheels for 3.10 on Linux, and instead
pipwill try to build from source.
3. Incompatible packages
Sometimes it’s just a matter of recompiling the code. That means you can can compile it yourself, with enough work, and the maintainer will usually release a new wheel pretty quickly. In other cases, however, the code is incompatible and requires some work to support new Python releases.
For example, when I first started writing this article I’d started porting the Fil memory profiler to 3.10, and initially the tests were failing.
Having fixed an incompatibility in the code, the next problem is that the test suite relies on the
numexpr package, and
numexpr can’t have 3.10 wheels until NumPy has complete 3.10 wheels.
So there’s a tree of dependencies blocking a new release.
As it turned:
- NumPy released a complete set of 3.10 wheel on Oct 20, 2021.
- Numexpr released 3.10 wheels on Dec 2, 2021.
- Fil will hopefully have a new release on Dec 2, 2021, if I have time.
Given many Python packages are maintained by volunteers with limited free time and working independently, it just takes time for updates to percolate through the system.
4. Lack of Conda support
Update: Conda-Forge did the initial 3.10 rollout, but actual package availability varies.
If you’re using Conda, Python 3.10 is not available yet. Conda-Forge has started the work, but the main Python package still hasn’t had a 3.10 release yet, and once it does all Python packages will need to be updated and rebuilt for 3.10.
5. Bugs in Python
Python 3.9.0 was released in October 2020. 3.9.1 was released 2 months later, with a long list of bugfixes.
Of course, there’s always another bug fix release in the future, but given a 12-month window between major releases, lots of new code will get written but not extensively used. Which means a follow-up release with plenty of bug fixes. Python 3.10.1 will be available Dec 6, 2021, and no doubt will have many bug fixes as libraries and then applications start switching and encountering more issues.
6. Lack of toolchain support
Update Nov 17, 2021: A new release of Black now supports 3.10 syntax.
New versions of Python often have new syntax, and that is the case with Python 3.10: you can use structural pattern matching. However, other tools need to support the new syntax too—autoformatters, linters, and so on. Until they do, you can’t use the new syntax.
For example, as I’m writing this the
black code formatter doesn’t yet support the pattern matching syntax.
What version should you be using?
Given that it takes work to upgrade—some additional testing, some tweaks to your code—it can be tempting to put off upgrading Python versions indefinitely. Why worry about incompatibilities, new versions, and what not, when you can just stick with your current version indefinitely?
The problem is that Python isn’t supported indefinitely, nor do libraries support all Python versions indefinitely. So if you’re running on a 5-year-old version of Python, switching becomes a Big Deal—you’ll often end up dealing with more significant cross-version changes in both Python and in libraries at the same time. Which makes upgrading scary. And sooner or later you will have to upgrade.
Python 3.6, for example, is going to end security updates in December 2021. It’s true, Linux distributions that ship it, like RHEL 8, will continue to provide security updates. But RHEL 8 also packages Python 3.8 and 3.9; you really shouldn’t be using 3.6 any more.
Instead of one massive scary upgrade event every few years, it’s much safer to have a continuous, ongoing process of smaller upgrades. Whenever a new major Python version comes out, or a new major library version, wait a bit, and then switch.
Which is to say that at the moment, you really should be using 3.9 if you can. If you’re far enough behind, do a series of upgrades: 3.7, 3.8, 3.9, and eventually, 3.10.
When should you switch to Python 3.10?
At a minimum, you will need to wait until:
- All your libraries explicitly support the new Python release.
- All the tools you rely on explicitly support the new Python release.
It’s probably worth waiting a couple months, until mid-December 2021, before even bothering to try your application. In addition to dependency availability and toolchain support, at that point you may also have access to the first point release, 3.10.1. If dependencies are still missing, keep trying again every month.
The concise and action-oriented guide to Docker packaging for production
Docker packaging for production is complicated, with as many as 70+ best practices to get right. And you want small images, fast builds, and your Python application running securely.
Take the fast path to learning best practices, by using the Python on Docker Production Handbook.
Free ebook: Introduction to Dockerizing for Production
Learn a step-by-step iterative DevOps packaging process in this free mini-ebook. You'll learn what to prioritize, the decisions you need to make, and the ongoing organizational processes you need to start.
Plus, you'll join my newsletter and get weekly articles covering practical tools and techniques, from Docker packaging to Python best practices.