When should you upgrade to Python 3.12?
Python 3.12 is out now–but should you switch to it immediately? And if you shouldn’t upgrade just yet, when should you?
Immediately after the release, you may not want to upgrade just yet. But from December 2023 and onwards, upgrading is definitely worth trying. To understand why, we need to consider Python packaging, the software development process, and take a look at the history of past releases.
The problems with a new major Python release
As with many open source projects, Python, Python libraries, and most of the toolchain and packages mentioned in this article are based on volunteer labor. None of this should be read as a complaint towards the people doing the maintenance, they’re doing hugely valuable work, released for free and often done for free, and everything takes time. In addition, because upgrades involve so many different groups, coordination and releases take even more time.
With that in mind, let’s consider the problems with using 3.12 immediately after a release, and in the following months:
1. Incompatible packages
In some cases, the code in a library is incompatible with a new Python release.
The Python 3.12 support for the Numba project, for example, is still a work-in-progress as of Oct 18, 2023. It’s far enough along to test, but it’s not out yet. PyTorch is also a work in progress. For my own projects, some should just work, but others I haven’t gotten to as of mid-October.
Even if the code is compatible, if dependencies aren’t compatible that can still delay a release. For example, the initial blocker for doing a 3.12 release for my Rust-based projects was PyO3 doing a new release. Now that they’ve done that, I’m working through my backlog.
Given many Python packages are maintained by volunteers with limited free time and working independently, it just takes time for updates to percolate through the system.
2. Missing binary packages
Even if the code is compatible, not all packages can be easily installed on 3.12 yet.
Typically, Python package maintainers upload compiled versions of their packages–known as “wheels”–to PyPI. When you install a package, you can just download the binary wheel and don’t need to compile it (unless you’re using Alpine Linux, in which case wheels may or may not be available).
But at least for a short time after 3.12’s release, some packages won’t have wheels for Python 3.12 uploaded yet. This is less of a problem these days compared to older Python releases, as:
- Package maintainers are putting in more and more effort to shorten this window, often releasing updated packages in advance of the Python final release.
- Some packages release binary wheels using the forward-compatible “abi3” format, which means newer Python releases just work with no extra effort.
Nonetheless, there will be packages that won’t have binary wheels yet. Best case, that means installation will take a lot longer as you have to compile the code on installation. Worst case, compilation will fail because you don’t have relevant compile-time dependencies installed.
Important: Make sure you upgrade pip before using it. Older versions might not support the binary wheels for 3.11 on Linux, and instead
pipwill try to build from source.
If you’re using Conda-Forge, the Python 3.12 rollout happens pretty quickly. A lot of it is automated, one of the benefits of Conda-Forge over PyPI, so as long as the underlying packages already support 3.12, it happens pretty quickly across the whole package repository.
3. Bugs in Python
Python 3.11.0 was released in October 2022. 3.11.1 was released in December 2022, with a long list of bugfixes.
Of course, there’s always another bug fix release in the future, but given a 12-month window between major releases, lots of new code will get written but not extensively used. Which means a follow-up release with plenty of bug fixes.
Things are getting better!
I’ve been updating this article over the years since 3.9, and the situation has gotten significantly better since then. Every release I’ve reduced the caveats, and projects have sped up their iteration cycles so they have support for newer releases even faster.
Is upgrading worth it?
Given that it takes work to upgrade—some additional testing, some tweaks to your code—it can be tempting to put off upgrading Python versions indefinitely. Why worry about incompatibilities, new versions, and what not, when you can just stick with your current version indefinitely?
The problem is that Python isn’t supported indefinitely, nor do libraries support all Python versions indefinitely; eventually you’ll stop getting security updates and bug fixes. Python 3.7 ended security updates in June 2023. 3.8 will be next, dropping security updates in October 2024.
So sooner or later you will have to upgrade. And if you’re running on a 5-year-old version of Python, switching becomes a Big Deal—you’ll often end up dealing with more significant cross-version changes in both Python and in libraries at the same time. Which makes upgrading scary.
Instead of one massive scary upgrade event every few years, it’s much safer to have a continuous, ongoing process of smaller upgrades. Whenever a new major Python version comes out, or a new major library version, wait a bit, and then switch.
The week Python 3.12 was released, you probably didn’t want to upgrade, but you certainly wanted to be using 3.11. If you’re far enough behind, do a series of upgrades: 3.9, 3.10, 3.11, and eventually, 3.12.
When should you switch your application to Python 3.12?
At a minimum, after a major new release you will need to wait until:
- All your libraries explicitly support the new Python release.
- All the tools you rely on explicitly support the new Python release.
Personally I’d start looking into it in December 2023, after 3.12.1 is out. If the dependencies you require are still missing 3.12 support, keep trying again every month.
The concise and action-oriented guide to Docker packaging for production
Docker packaging for production is complicated, with as many as 70+ best practices to get right. And you want small images, fast builds, and your Python application running securely.
Take the fast path to learning best practices, by using the Python on Docker Production Handbook.
Free ebook: "Introduction to Dockerizing for Production"
Learn a step-by-step iterative DevOps packaging process in this free mini-ebook. You'll learn what to prioritize, the decisions you need to make, and the ongoing organizational processes you need to start.
Plus, you'll join over 7400 people getting weekly emails covering practical tools and techniques, from Docker packaging to Python best practices.