Speed up your code

Table of Contents

What does speed mean?

  1. Transgressive Programming: the magic of breaking abstraction boundaries
    Much of the time you need to stick to abstraction boundaries when writing code. But when to comes to domains like performance or debugging, you need to deliberately break those boundaries.

  2. Optimizing your code is not the same as parallelizing your code
    To make your Python code faster, you should often start with optimizing single-threaded versions, then consider multiprocessing, and only then think about a cluster.

  3. Speed is situational: two websites, two orders of magnitude
    How do you make your application fast? It depends every much on your particular case, as you’ll see in this example case study.

Measuring performance

  1. Where’s your bottleneck? CPU time vs wallclock time
    Slow runtime isn’t necessarily CPU: other causes include I/O, locks, and more. Learn a quick heuristic to help you identify which it is.

  2. Beyond cProfile: Choosing the right tool for performance optimization
    There are different profilers you can use to measure Python performance. Learn about cProfile, sampling profilers, and logging, and when to use each.
    (Originally a PyGotham 2019 talk—you can also watch a video)

  3. Not just CPU: writing custom profilers for Python
    Sometimes existing profilers aren’t enough: you need to measure something unusual. With Python’s built-in profiling library, you can write your own.

  4. CI for performance: Reliable benchmarking in noisy environments
    Running performance benchmarks can result in noise drowning out the signal. Learn how to get reliable performance benchmarks with Cachegrind.

  5. Logging for scientific computing: debugging, performance, trust
    Logging is a great way to understand your code, make it faster, and help you convince yourself and others that you can trust the results. Learn why, with examples built on the Eliot logging library.
    (Originally a PyCon 2019 talk—you can also watch a video)

Multiprocessing

  1. Why your multiprocessing Pool is stuck (it’s full of sharks!)
    Python’s multiprocessing library is an easy way to use multiple CPUs, but its default configuration can lead to deadlocks and brokenness. Learn why, and how to fix it.

  2. The Parallelism Blues: when faster code is slower
    By default NumPy uses multiple CPUs for certain operations. But sometimes that can actually slow down your code.

Libraries and applications

  1. Choosing a faster JSON library for Python
    There are multiple JSON encoding/decoding libraries available for Python. Learn how you can choose the fastest for your particular use case.

  2. All Pythons are slow, but some are faster than others
    Python on Ubuntu is not the same speed as Python in the python Docker image. So I ran some benchmarks, so you can pick the fastest.

Speed up your testing

  1. When C extensions crash: easier debugging for your Python application
    If your test suite crashes half-way through, debugging is quite difficult if it’s a crash in C code. Learn how to setup CI in advance to help you quickly figure out what’s going wrong.

  2. Farewell to fsync(): 10× faster database tests with Docker
    Realistic tests require using a real database—but that can be slow. Luckily, production databases have constraints you don’t have when testing, allowing you to run them much faster.

  3. When your CI is taking forever on AWS EC2, it might be EBS
    Learn an unexpected cause for slowness on EC2, and what you can do about.

  4. Stuck with slow tests? Speed up your feedback loop
    Sometimes you can’t speed up your test suite. What you can do, however, is find failures faster.

  5. Why Pylint is both useful and unusable, and how you can actually use it
    Ideally you will find bugs in your code before you even run it; as you’re typing, even. PyLint is a great tool for this, but it also has some problems you’ll need to work around.

  6. Fast tests for slow services: why you should use verified fakes
    Sometimes your tests need to talk to something slow and not under your control. A common solution is Mocks, but those can be unrealistic. Instead, you can use verified fakes, which try to be both fast and realistic.



Learn practical Python software engineering skills you can use at your job

Too much to learn? Don't know where to start?

Sign up for my newsletter, and join over 2800 Python developers and data scientists learning practical tools and techniques, from Docker packaging to testing to Python best practices, with a free new article in your inbox every week.