When to switch to Python 3.9
Python 3.9 is now available–but should you switch to it immediately? And if not now, when?
The short answer is, no, you probably don’t want to switch immediately; quite possibly you can’t switch immediately. To understand why, we need to consider Python packaging, the software development process, and take a look at the history of past releases.
We can then make a guess about when Python 3.9 will actually be usable.
The problems with a new major Python release
I’m writing this article on October 6th, 2020, the day after 3.9 was released… and far too early to start using Python 3.9. Let’s see why.
1. Missing Docker images
Let’s try running Python 3.9 under Docker:
$ docker run -it python:3.9 Unable to find image 'python:3.9' locally docker: Error response from daemon: manifest for python:3.9 not found: manifest unknown: manifest unknown.
As it turns out, there is no Python 3.9 Docker image for the final release.
If you want to use Python 3.9, you’re stuck with release candidates, e.g.
python:3.9-rc-slim will install 3.9.0-rc2.
No doubt it will be available within a week, but early on even installing a new release can be difficult. And that’s just our first problem.
2. Missing packages
Next, let’s try installing some packages. Pandas is a pretty popular package; it was downloaded 25 million times from PyPI over the last month. When I try to install it on Python 3.9 things don’t go well:
$ docker run python:3.9-rc-slim pip install pandas ... File "numpy/core/setup.py", line 669, in get_mathlib_info raise RuntimeError("Broken toolchain: cannot link a simple C program") RuntimeError: Broken toolchain: cannot link a simple C program ---------------------------------------- ERROR: Command errored out with exit status 1: /usr/local/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv = '"'"'/tmp/pip-install-0aps2yxr/numpy/setup.py'"'"'; __file__='"'"'/tmp/pip-install-0aps2yxr/numpy/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-tl7_htm8/install-record.txt --single-version-externally-managed --prefix /tmp/pip-build-env-0zc3r6es/overlay --compile --install-headers /tmp/pip-build-env-0zc3r6es/overlay/include/python3.9/numpy Check the logs for full command output. ---------------------------------------- ERROR: Command errored out with exit status 1: /usr/local/bin/python /usr/local/lib/python3.9/site-packages/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-0zc3r6es/overlay --no-warn-script-location --no-binary :none: --only-binary :none: -i https://pypi.org/simple -- setuptools wheel 'Cython>=0.29.21,<3' 'numpy==1.15.4; python_version=='"'"'3.6'"'"' and platform_system!='"'"'AIX'"'"'' 'numpy==1.15.4; python_version=='"'"'3.7'"'"' and platform_system!='"'"'AIX'"'"'' 'numpy==1.17.3; python_version>='"'"'3.8'"'"' and platform_system!='"'"'AIX'"'"'' 'numpy==1.16.0; python_version=='"'"'3.6'"'"' and platform_system=='"'"'AIX'"'"'' 'numpy==1.16.0; python_version=='"'"'3.7'"'"' and platform_system=='"'"'AIX'"'"'' 'numpy==1.17.3; python_version>='"'"'3.8'"'"' and platform_system=='"'"'AIX'"'"'' Check the logs for full command output.
What’s going on?
Typically, Python package maintainers upload compiled versions of their packages–known as “wheels”–to PyPI. When you install a package, you can just download the binary wheel and don’t need to compile it (unless you’re using Alpine Linux).
But it’s so soon after 3.9’s release, many packages don’t have wheels for Python 3.9 yet. For example, as of October 6th neither the PyPI file listing for NumPy (a Pandas dependency) nor the file listing for Pandas show Python 3.9 wheels. You can, of course, compile these packages from source, assuming they work out of the box with Python 3.9–but that takes a bunch more effort (setting up compilers) and time.
Some packages don’t even distribute source packages on PyPI. For example, my Fil memory profiler, or on a much more significant scale the very popular opencv-python package whose whole purpose is distributing binaries.
When Python 3.8.0 first came out, it took a month before
opencv-python had a wheel available.
3. Bugs in Python
Python 3.8.0 was released in October 2019. 3.8.1 was released 2 months later, with a long list of bugfixes.
Of course, there’s always another bug fix release in the future, but given an 12-month window between major releases, lots of new code will get written but not extensively used. Which means a follow-up release with plenty of bug fixes. Python 3.9.1 will likely have many bug fixes as libraries and then applications start switching and encountering more issues.
4. Lack of toolchain support
New versions of Python often have new syntax, and that is the case with Python 3.9: you can use arbitrary expressions in decorators. However, other tools need to support the new syntax too—autoformatters, linters, and so on. Until they do, you can’t use the new syntax.
For example, as I’m writing this the released version of the
black code formatter doesn’t support 3.9 yet.
Why you should switch
Given that it takes work to upgrade—some additional testing, some tweaks to your code—it can be tempting to put off switching indefinitely. Why worry about incompatibilities, new versions, and what not, when you can just stick with your current version indefinitely?
The problem is that Python isn’t supported indefinitely, nor do libraries support all Python versions indefinitely. So if you’re running on a 5-year-old version of Python, switching becomes a Big Deal—you’ll often end up dealing with more significant cross-version changes in both Python and in libraries at the same time. Which makes upgrading scary.
Instead of one massive scary upgrade event every few years, it’s much safer to have a continuous, ongoing process of smaller upgrades. Whenever a new major Python version comes out, or a new major library version, wait a bit, and then switch.
When should you switch to Python 3.9?
At a minimum, you will need to wait until:
- All your libraries explicitly support the new Python release.
- All the tools you rely on explicitly support the new Python release.
Wait a couple months, until mid-December 2020, before even bothering to try your application. In addition to dependency availability and toolchain support, at that point you may also have access to the first point release, 3.9.1. If dependencies are still missing, keep trying again every month.
Learn how to build fast, production-ready Docker images—read the rest of the Docker packaging guide for Python.
Production Docker packaging is too complicated to learn from Google searches
With as much as a dozen different intersecting technologies, and an unknown number of details to get right, Docker packaging isn't simple, especially for production.
But you still need fast builds that save you time, and security best practices that keep you safe.
Take the fast path to learning best practices, by using the Python on Docker Production Handbook.
Free ebook: Introduction to Dockerizing for Production
Learn a step-by-step iterative DevOps packaging process in this free mini-ebook. You'll learn what to prioritize, the decisions you need to make, and the ongoing organizational processes you need to start.
Plus, you'll join my newsletter and get weekly articles covering practical tools and techniques, from Docker packaging to Python best practices.