Speed up your code

Table of Contents

What does speed mean?

  1. Transgressive Programming: the magic of breaking abstractions
    Usually you want to stick to abstraction boundaries when coding. But for performance or debugging, you may need to deliberately break those boundaries.

  2. Speed is situational: two websites, two orders of magnitude
    How do you make your application fast? It depends every much on your particular case, as you’ll see in this example case study.

  3. Optimizing your code is not the same as parallelizing your code
    To make your Python code faster, start with optimizing single-threaded versions, then consider multiprocessing, and only then think about a cluster.

Measuring performance

  1. Where’s your bottleneck? CPU time vs wallclock time
    Slow software performance may be due to CPU, I/O, locks, and more. Learn a quick heuristic to help you identify which it is.

  2. Beyond cProfile: Choosing the right tool for performance optimization
    There are different profilers you can use to measure Python performance. Learn about cProfile, sampling profilers, and logging, and when to use each.

  3. Not just CPU: writing custom profilers for Python
    Sometimes existing Python profilers aren’t enough: you need to measure something unusual. Learn how to write your own cProfile-based custom profiler.

  4. Logging for scientific computing: debugging, performance, trust
    Logging can help you understand and speed up your scientific computing code, and convince yourself and others that you can trust the results.
    (Originally a PyCon 2019 talk—you can also watch a video)

  5. CI for performance: Reliable benchmarking in noisy environments
    Running performance benchmarks can result in noise drowning out the signal. Learn how to get reliable performance benchmarks with Cachegrind.

  6. Docker can slow down your code and distort your benchmarks
    In theory, Docker containers have no performance overhead. In practice, they can actually slow down your code and distort performance measurements.

Multiprocessing

  1. Why your multiprocessing Pool is stuck (it’s full of sharks!)
    On Linux, the default configuration of Python’s multiprocessing library can lead to deadlocks and brokenness. Learn why, and how to fix it.

  2. The Parallelism Blues: when faster code is slower
    By default NumPy uses multiple CPUs for certain operations. But sometimes parallelism can actually slow down your code.

Libraries and applications

  1. Choosing a faster JSON library for Python
    There are multiple JSON encoding/decoding libraries available for Python. Learn how you can choose the fastest for your particular use case.

  2. All Pythons are slow, but some are faster than others
    Python on Ubuntu is not the same speed as Python in the python Docker image. So I ran some benchmarks, so you can pick the fastest.

  3. The hidden performance overhead of Python C extensions
    A compiled language like Rust or C is a lot faster than Python, with some caveats. Learn about the hidden overhead you’ll need to overcome.

  4. Cython, Rust, and more: choosing a language for Python extensions
    You can write Python extensions with Cython, Rust, and many other tools. Learn which one you should use, depending on your particular needs.

Speed up your testing

  1. Fast tests for slow services: why you should use verified fakes
    Sometimes your Python tests need to talk to something slow and not under your control. Learn how to write fast and realistic tests, without resorting to mocks.

  2. Why Pylint is both useful and unusable, and how you can use it
    You want to find bugs in your Python code before as you write your code. PyLint is a great tool for this, but it has some problems you’ll need to work around.

  3. Stuck with slow tests? Speed up your feedback loop
    Sometimes you can’t speed up your Python test suite. What you can do, however, is find failures faster with linters, partial testing, and more.

  4. When your CI is taking forever on AWS, it might be EBS
    When running tests or builds on AWS, a bad EBS configuration can slow everything down; learn how to identify the problem and speed up your build.

  5. Farewell to fsync(): 10× faster database tests with Docker
    Realistic tests require a real database—but that can be slow. Luckily, testing has different constraints than production, allowing you to run them much faster.

  6. When C extensions crash: easier debugging for your Python application
    If your Python test suite segfaults in C code, debugging is difficult. But an easy configuration tweak can help you pinpoint the responsible code.



Learn practical Python software engineering skills you can use at your job

Too much to learn? Don't know where to start?

Sign up for my newsletter, and join over 4600 Python developers and data scientists learning practical tools and techniques, from Docker packaging to testing to Python best practices, with a free new article in your inbox every week.