Speed up your code
Table of Contents
- What does speed mean?
- Measuring performance
- Libraries and applications
- Speed up your testing
What does speed mean?
Transgressive Programming: the magic of breaking abstraction boundaries
Much of the time you need to stick to abstraction boundaries when writing code. But when to comes to domains like performance or debugging, you need to deliberately break those boundaries.
Optimizing your code is not the same as parallelizing your code
To make your Python code faster, you should often start with optimizing single-threaded versions, then consider multiprocessing, and only then think about a cluster.
Speed is situational: two websites, two orders of magnitude
How do you make your application fast? It depends every much on your particular case, as you’ll see in this example case study.
Where’s your bottleneck? CPU time vs wallclock time
Slow runtime isn’t necessarily CPU: other causes include I/O, locks, and more. Learn a quick heuristic to help you identify which it is.
Beyond cProfile: Choosing the right tool for performance optimization
There are different profilers you can use to measure Python performance. Learn about cProfile, sampling profilers, and logging, and when to use each.
Not just CPU: writing custom profilers for Python
Sometimes existing profilers aren’t enough: you need to measure something unusual. With Python’s built-in profiling library, you can write your own.
CI for performance: Reliable benchmarking in noisy environments
Running performance benchmarks can result in noise drowning out the signal. Learn how to get reliable performance benchmarks with Cachegrind.
Logging for scientific computing: debugging, performance, trust
Logging is a great way to understand your code, make it faster, and help you convince yourself and others that you can trust the results. Learn why, with examples built on the Eliot logging library.
(Originally a PyCon 2019 talk—you can also watch a video)
Docker can slow down your code and distort your benchmarks
In theory, containers have no performance overhead. In practice, they can actually slow down your code and distort your performance measurements.
Why your multiprocessing Pool is stuck (it’s full of sharks!)
Python’s multiprocessing library is an easy way to use multiple CPUs, but its default configuration can lead to deadlocks and brokenness. Learn why, and how to fix it.
The Parallelism Blues: when faster code is slower
By default NumPy uses multiple CPUs for certain operations. But sometimes that can actually slow down your code.
Libraries and applications
Choosing a faster JSON library for Python
There are multiple JSON encoding/decoding libraries available for Python. Learn how you can choose the fastest for your particular use case.
The hidden performance overhead of Python C extensions
A compiled language like Rust or C is a lot faster than Python, but it won’t always make your Python code faster. Learn about the hidden overhead you’ll need to overcome.
All Pythons are slow, but some are faster than others
Python on Ubuntu is not the same speed as Python in the
pythonDocker image. So I ran some benchmarks, so you can pick the fastest.
Speed up your testing
When C extensions crash: easier debugging for your Python application
If your test suite crashes half-way through, debugging is quite difficult if it’s a crash in C code. Learn how to setup CI in advance to help you quickly figure out what’s going wrong.
Farewell to fsync(): 10× faster database tests with Docker
Realistic tests require using a real database—but that can be slow. Luckily, production databases have constraints you don’t have when testing, allowing you to run them much faster.
When your CI is taking forever on AWS EC2, it might be EBS
Learn an unexpected cause for slowness on EC2, and what you can do about.
Stuck with slow tests? Speed up your feedback loop
Sometimes you can’t speed up your test suite. What you can do, however, is find failures faster.
Why Pylint is both useful and unusable, and how you can actually use it
Ideally you will find bugs in your code before you even run it; as you’re typing, even. PyLint is a great tool for this, but it also has some problems you’ll need to work around.
Fast tests for slow services: why you should use verified fakes
Sometimes your tests need to talk to something slow and not under your control. A common solution is Mocks, but those can be unrealistic. Instead, you can use verified fakes, which try to be both fast and realistic.
Learn practical Python software engineering skills you can use at your job
Too much to learn? Don't know where to start?
Sign up for my newsletter, and join over 3700 Python developers and data scientists learning practical tools and techniques, from Docker packaging to testing to Python best practices, with a free new article in your inbox every week.