What’s using up all your memory?
You need to massively reduce the memory usage of your Python data processing code. And you don’t know which part of your code is responsible.
tracemalloc and your code doesn’t even show up.
Next you try
memory_profiler and it kinda sorta works—if you have all day to manually annotate each function recursively.
And you still don’t really understand where the memory is going.
Existing tools are slow, error prone, and often insufficient—and if you can’t measure memory usage, it’s very hard to optimize your code.
You want to know where your program’s peak memory usage is coming from, and you want to be able to tell at a glance.
Fil: a better memory profiler for Python
Fil is a new Python memory profiler, designed for code that processes large amounts of data: the kind of code written by data scientists, data engineers, and scientists.
Instead of measuring random snapshots in time, Fil figures out your bottleneck: the moment in time your memory usage is highest. And then it tells you where exactly all that memory was allocated.
Here’s what the output looks like—the width of the frames shows you what percentage of memory that function was responsble for.
Notice how it’s visually clear that most of the peak memory usage was allocated by
register_translation() and the functions it calls.
If you needed to optimize the memory usage of this program, Fil would tell you exactly where to start.
You can hover your mouse over the frames to get more information; you can also click on a frame (e.g.
register_translation) to get a zoomed in view of that part of the flamegraph.
Need to reduce memory usage in your Python program?
As of February 2020, Fil is still in the alpha testing stage, so it’s not yet generally available. Sign up below to learn when it becomes available and get a chance to try it—at some point soon I will be looking for beta testers.
Be the first to learn when Fil is available
Get on the beta list, so you can try Fil for yourself.
Plus, get weekly emails showing you how to reduce memory usage, optimize your code, and improve your software engineering skills with Python, Docker, and more: