Speed up your code
Table of Contents
- What does speed mean?
- Measuring performance
- Libraries and applications
- Speed up your testing
What does speed mean?
Transgressive Programming: the magic of breaking abstractions
Usually you want to stick to abstraction boundaries when coding. But for performance or debugging, you may need to deliberately break those boundaries.
Optimizing your code is not the same as parallelizing your code
To make your Python code faster, start with optimizing single-threaded versions, then consider multiprocessing, and only then think about a cluster.
Speed is situational: two websites, two orders of magnitude
How do you make your application fast? It depends every much on your particular case, as you’ll see in this example case study.
Where’s your bottleneck? CPU time vs wallclock time
Slow software performance may be due to CPU, I/O, locks, and more. Learn a quick heuristic to help you identify which it is.
Beyond cProfile: Choosing the right tool for performance optimization
There are different profilers you can use to measure Python performance. Learn about cProfile, sampling profilers, and logging, and when to use each.
Not just CPU: writing custom profilers for Python
Sometimes existing Python profilers aren’t enough: you need to measure something unusual. Learn how to write your own cProfile-based custom profiler.
CI for performance: Reliable benchmarking in noisy environments
Running performance benchmarks can result in noise drowning out the signal. Learn how to get reliable performance benchmarks with Cachegrind.
Logging for scientific computing: debugging, performance, trust
Logging can help you understand and speed up your scientific computing code, and convince yourself and others that you can trust the results.
(Originally a PyCon 2019 talk—you can also watch a video)
Docker can slow down your code and distort your benchmarks
In theory, Docker containers have no performance overhead. In practice, they can actually slow down your code and distort performance measurements.
Why your multiprocessing Pool is stuck (it’s full of sharks!)
On Linux, the default configuration of Python’s multiprocessing library can lead to deadlocks and brokenness. Learn why, and how to fix it.
The Parallelism Blues: when faster code is slower
By default NumPy uses multiple CPUs for certain operations. But sometimes parallelism can actually slow down your code.
Libraries and applications
Choosing a faster JSON library for Python
There are multiple JSON encoding/decoding libraries available for Python. Learn how you can choose the fastest for your particular use case.
The hidden performance overhead of Python C extensions
A compiled language like Rust or C is a lot faster than Python, with some caveats. Learn about the hidden overhead you’ll need to overcome.
All Pythons are slow, but some are faster than others
Python on Ubuntu is not the same speed as Python in the
pythonDocker image. So I ran some benchmarks, so you can pick the fastest.
Speed up your testing
When C extensions crash: easier debugging for your Python application
If your Python test suite segfaults in C code, debugging is difficult. But an easy configuration tweak can help you pinpoint the responsible code.
Farewell to fsync(): 10× faster database tests with Docker
Realistic tests require a real database—but that can be slow. Luckily, testing has different constraints than production, allowing you to run them much faster.
When your CI is taking forever on AWS, it might be EBS
When running tests or builds on AWS, a bad EBS configuration can slow everything down; learn how to identify the problem and speed up your build.
Stuck with slow tests? Speed up your feedback loop
Sometimes you can’t speed up your Python test suite. What you can do, however, is find failures faster with linters, partial testing, and more.
Why Pylint is both useful and unusable, and how you can use it
You want to find bugs in your Python code before as you write your code. PyLint is a great tool for this, but it has some problems you’ll need to work around.
Fast tests for slow services: why you should use verified fakes
Sometimes your Python tests need to talk to something slow and not under your control. Learn how to write fast and realistic tests, without resorting to mocks.
Learn practical Python software engineering skills you can use at your job
Too much to learn? Don't know where to start?
Sign up for my newsletter, and join over 4100 Python developers and data scientists learning practical tools and techniques, from Docker packaging to testing to Python best practices, with a free new article in your inbox every week.