Easy Python memory profiling for data scientists and scientists with Fil
Your Python code reads some data, processes it, and uses too much memory; maybe it even dies due to an out-of-memory error. In order to reduce memory usage, you first need to figure out:
- Where peak memory usage is, also known as the high-water mark.
- What code was responsible for allocating the memory that was present at that peak moment.
That’s exactly what the Fil memory profiler for Python will do, with support for both Linux and macOS.
“Within minutes of using your tool, I was able to identify a major memory bottleneck that I never would have thought existed. The ability to track memory allocated via the Python interface and also C allocation is awesome, especially for my NumPy / Pandas programs.”
—Derrick Kondo
- Read the documentation:
- Visit the GitHub page.
Performance profiling and insights into production with Sciagraph
Fil is designed for offline development, and only does memory profiling. It does not support multiprocessing.
But you can get both memory and performance profiling using my other project, the Sciagraph profiler. While it is a commercial project, it comes with a free plan. It has much lower overhead, supports multiprocessing, and can optionally be used in production as well.
Here’s a quick comparison:
| Fil (Open Source) | Sciagraph (Commercial, with free plan) | |
|---|---|---|
| Memory profiling | ✅ — all allocations | ✅ — sampled |
| Performance profiling | ❌ | ✅ |
| Multiprocessing support | ❌ | ✅ |
| Runtime overhead | Up to 40-50% | Negligible to ~5% |
| Use in production? | ❌, too slow | ✅ (with some paid plans) |
| Supported platforms | Linux, macOS | Linux, macOS |
Learn more about Sciagraph here.
Deep insight with minimal effort
Fil is designed to be:
- Easy to use: It has as few command-line options and knobs as possible.
- Easy to understand: The output has been tuned to direct your attention towards the bottlenecks.
With just a simple change to how you run your Python program, you can learn exactly where memory allocations are coming from, specifically for peak memory usage. Just focus on the wider, redder frames to get a stack trace of the code responsible for peak memory usage:
More user feedback
“Fil has just pointed straight at the cause of a memory issue that’s been costing my team tons of time and compute power. Thanks again for such an excellent tool!”
—Peter Sobot
Robot Framework 4.1.2 uses considerably less memory than earlier versions… Memory usage was profiled using the Fil tool. It made it easy to see that memory usage had increased and, more importantly, where memory was spent. The latter allowed making small changes in few places to drop memory usage considerably.