Beyond cProfile @ PyGotham 2019

Talk slides

You can find the slides for the talk here. I hope to have a prose version available soon as well.

Learn more

If you want to learn more about Python performance, you may also enjoy these articles:

• Why your multiprocessing Pool is stuck (it’s full of sharks!)

• Choosing a faster JSON library for Python

• Logging for scientific computing: debugging, performance, trust

• Speed is situational: two websites, two orders of magnitude

• Where’s your bottleneck? CPU time vs wallclock time

• Not just CPU: writing custom profilers for Python

• Beyond cProfile: Choosing the right tool for performance optimization

• The Parallelism Blues: when faster code is slower

• Optimizing your code is not the same as parallelizing your code

• CI for performance: Reliable benchmarking in noisy environments

• All Pythons are slow, but some are faster than others

• Transgressive Programming: the magic of breaking abstractions

• The hidden performance overhead of Python C extensions

• Docker can slow down your code and distort your benchmarks

• Cython, Rust, and more: choosing a language for Python extensions

• Creating a better flamegraph visualization

• Memory location matters for performance

• How vectorization speeds up your Python code

• The fastest way to read a CSV in Pandas

• Faster Python calculations with Numba: 2 lines of code, 13× speed-up

• Faster hardware is a bad first solution to slow software

• When Python can’t thread: a deep-dive into the GIL’s impact

• Faster, more memory-efficient Python JSON parsing with msgspec

• CPUs, cloud VMs, and noisy neighbors: the limits of parallelism

• Pandas vectorization: faster code, slower code, bloated memory

• Finding performance bottlenecks in Celery tasks

• The limits of Python vectorization as a performance technique

• The best way to find performance bottlenecks: observing production

• Finding performance problems: profiling or logging?

• Invasive procedures: Python affordances for performance measurement

• Find slow data processing tasks (before your customers do)

• Early speed optimizations aren’t premature

• Good old-fashioned code optimization never goes out of style

• Who controls parallelism? A disagreement that leads to slower code

• Some reasons to avoid Cython

• Python’s multiprocessing performance problem

• Speeding up text processing in Python (is hard)

• Polars for initial data analysis, Polars for production

• Choosing a good file format for Pandas

• Understanding CPUs can help speed up Numba and NumPy code

• When NumPy is too slow

• The easiest way to speed up Python with Rust

• Speeding up your code when multiple cores aren’t an option

• Speeding up Cython with SIMD

• Two kinds of threads pools, and why you need both

• Using Polars in a Pandas world

• How many CPU cores can you actually use in parallel?

• NumPy 2 is coming: preventing breakage, updating your code

• Beware of misleading GPU vs CPU benchmarks

• Profiling your Numba code

• Not just NVIDIA: GPU programming that runs everywhere

• Let’s optimize! Running 15× faster with a situation-specific algorithm

• Let’s build and optimize a Rust extension for Python

You might also benefit from my guide to production-ready Docker packaging for Python.

Training

If you enjoyed my talk and would like your team to improve their skills so they can ship features faster, perhaps you could benefit from my training classes.