Fil: A memory profiler for Python on Linux and macOS

Your Python code reads some data, processes it, and uses too much memory; maybe it even dies due to an out-of-memory error. In order to reduce memory usage, you first need to figure out:

Where peak memory usage is, also known as the high-water mark.
What code was responsible for allocating the memory that was present at that peak moment.

That’s exactly what Fil will help you find. Fil an open source memory profiler designed for data processing applications written in Python, and includes native support for Jupyter.

Fil is open source, and is designed for offline profiling. It has enough of a performance impact that you won’t want to use it on production workloads, but it can profile even small amounts of memory.

If you want memory (and performance!) profiling for your Python batch jobs in production, consider using Sciagraph.

Getting started with Fil

In this section you’ll learn how to:

Installing Fil

Fil requires macOS or Linux, and Python 3.7 or Later. You can either use Conda, a sufficiently new version of Pip, or higher-level tools like Poetry or Pipenv.

Conda

To install on Conda:

$ conda install -c conda-forge filprofiler

Pip (or similar tools)

To install the latest version of Fil you’ll need Pip 19 or newer. You can check the current version like this:

$ pip --version
pip 19.3.0

If you’re using something older than v19, you can upgrade by doing:

$ pip install --upgrade pip

If that doesn’t work, try running your code in a virtualenv (always a good idea in general):

$ python3 -m venv venv/
$ source venv/bin/activate
(venv) $ pip install --upgrade pip

Assuming you have a new enough version of pip, you can now install Fil:

$ pip install filprofiler

Using Fil for the first time

First, install Fil.

Then, create a Python file called example.py with the following code:

import numpy as np

def make_big_array():
    return np.zeros((1024, 1024, 50))

def make_two_arrays():
    arr1 = np.zeros((1024, 1024, 10))
    arr2 = np.ones((1024, 1024, 10))
    return arr1, arr2

def main():
    arr1, arr2 = make_two_arrays()
    another_arr = make_big_array()

main()

Now, you can run it with Fil:

$ fil-profile run example.py

This will run the program under Fil, and pop up the results.

In the next section, we’ll look at the results and see what they tell us.

Understanding Fil’s output

Let’s look at the result of the Fil run from the previous section:

What does this mean?

What you’re seeing is a flamegraph, a visualization that shows a tree of callstacks and which ones were most expensive. In Fil’s case, it shows the callstacks responsible for memory allocations at the point in time when memory usage was highest.

The wider or redder the frame, the higher percentage of memory that function was responsible for. Each line is an additional call in the callstack.

This particular flamegraph is interactive:

Click on a frame to see a zoomed in view of that part of the callstack. You can then click “Reset zoom” in the upper left corner to get back to the main overview.
Hover over a frame with your mouse to get additional details.

To optimize your code, focus on the wider and redder frames. These are the frames that allocated most of the memory. In this particular example, you can see that the most memory was allocated by a line of code in the make_big_array() function.

Having found the source of the memory allocations at the moment of peak memory usage, you can then go and reduce memory usage. You can then validate your changes reduced memory usage by re-running your updated program with Fil and comparing the result.

Understanding Fil

In this section you’ll learn:

Fil vs other Python memory tools

There are two distinct patterns of Python usage, each with its own source of memory problems.

In a long-running server, memory usage can grow indefinitely due to memory leaks. That is, some memory is not being freed.

If the issue is in Python code, tools like tracemalloc and Pympler can tell you which objects are leaking and what is preventing them from being leaked.
If you’re leaking memory in C code, you can use tools like Valgrind.

Fil, however, is not specifically aimed at memory leaks, but at the other use case: data processing applications. These applications load in data, process it somehow, and then finish running.

The problem with these applications is that they can, on purpose or by mistake, allocate huge amounts of memory. It might get freed soon after, but if you allocate 16GB RAM and only have 8GB in your computer, the lack of leaks doesn’t help you.

Fil will therefore tell you, in an easy to understand way:

Where peak memory usage is, also known as the high-water mark.
What code was responsible for allocating the memory that was present at that peak moment.
This includes C/Fortran/C++/whatever extensions that don’t use Python’s memory allocation API (tracemalloc only does Python memory APIs).

This allows you to optimize that code in a variety of ways.

How Fil works

Fil uses the LD_PRELOAD/DYLD_INSERT_LIBRARIES mechanism to preload a shared library at process startup. This is why Fil can’t be used as regular library and needs to be started in a special way: it requires setting up the correct environment before Python starts.

This shared library intercepts all the low-level C memory allocation and deallocation API calls, and keeps track of the corresponding allocation. For example, instead of a malloc() memory allocation going directly to your operating system, Fil will intercept it, keep note of the allocation, and then call the underlying implementation of malloc().

At the same time, the Python tracing infrastructure (the same infrastructure used by cProfile and coverage.py) is used to figure out which Python callstack/backtrace is responsible for each allocation.

How to use Fil

In this section you will learn how to use Fil to profile:

Complete Python programs.
Cells in Jupyter notebooks.
Parts of your program using Fil’s Python API.

You will also learn how to use Fil to debug:

Profiling complete Python programs

You want to get a memory profile of your Python program end-to-end, from when it starts running to when it finishes.

Profiling Python scripts

Let’s say you usually run your program like this:

$ python yourscript.py --input-file=yourfile

Just do:

$ fil-profile run yourscript.py --input-file=yourfile

And it will generate a report and automatically try to open it in for you in a browser. Reports will be stored in the fil-result/ directory in your current working directory.

You can also use this alternative syntax:

$ python -m filprofiler run yourscript.py --input-file=yourfile

Profiling Python modules (`python -m`)

If your program is usually run as a module:

$ python -m yourapp.yourmodule --args

You can run it with Fil like this:

$ fil-profile run -m yourapp.yourmodule --args

Or like this:

$ python -m filprofiler run -m yourapp.yourmodule --args

Profiling in Jupyter

To measure peak memory usage of some code in Jupyter you need to do three things:

Using Fil in Jupyter

1. Use “Python 3 with Fil” kernel

Jupyter notebooks run with a particular “kernel”, which most of the time just determines which programming language the notebook is using, like Python or R. Fil support in Jupyter requires a special kernel, so instead of using the “Python 3” kernel you’ll use the “Python 3 with Fil” kernel.

There are two ways to choose this kernel:

You can choose this kernel when you create a new notebook.
You can switch an existing notebook in the Kernel menu. There should be a “Change Kernel” option in there in both Jupyter Notebook and JupyterLab.

2. Load the extension

In one of the cells in your notebook, add this to the cell:

%load_ext filprofiler

3. Profiling a particular cell

You can now do memory profiles of particular cells by adding %%filprofile as the first line of the cell.

Load the extension by doing %load_ext filprofiler.
Add the %%filprofile magic to the top of the cell with the code you wish to profile.

An example

Here’s an example session:

Screenshot of JupyterLab session

Profiling a subset of your Python program

Sometimes you only want to profile your Python program part of the time. For this use case, Fil provides a Python API.

Important: This API turns profiling on and off for the whole process! If you want more fine grained profiling, e.g. per thread, please file an issue.

Using the Python API

1. Add profiling in your code

Let’s you have some code that does the following:

def main():
    config = load_config()
    result = run_processing(config)
    generate_report(result)

You only want to get memory profiling for the run_processing() call.

You can do so in the code like so:

from filprofiler.api import profile

def main():
    config = load_config()
    result = profile(lambda: run_processing(config), "/tmp/fil-result")
    generate_report(result)

You could also make it conditional, e.g. based on an environment variable:

import os
from filprofiler.api import profile

def main():
    config = load_config()
    if os.environ.get("FIL_PROFILE"):
        result = profile(lambda: run_processing(config), "/tmp/fil-result")
    else:
        result = run_processing(config)
    generate_report(result)

2. Run your script with Fil

You still need to run your program in a special way. If previously you did:

$ python yourscript.py --config=myconfig

Now you would do:

$ fil-profile python yourscript.py --config=myconfig

Notice that you’re doing fil-profile python, rather than fil-profile run as you would if you were profiling the full script. Only functions running for the duration of the filprofiler.api.profile() call will have memory profiling enabled, including of course the function you pass in. The rest of the code will run at (close) to normal speed and configuration.

Each call to profile() will generate a separate report. The memory profiling report will be written to the directory specified as the output destination when calling profile(); in or example above that was "/tmp/fil-result". Unlike full-program profiling:

The directory you give will be used directly, there won’t be timestamped sub-directories. If there are multiple calls to profile(), it is your responsibility to ensure each call writes to a unique directory.
The report(s) will not be opened in a browser automatically, on the presumption you’re running this in an automated fashion.

Debugging out-of-memory crashes using Fil

Note: Out-of-memory detection is currently disabled on macOS. This may change in a future release.

Typically when your program runs out of memory, it will crash, or get killed mysteriously by the operating system, or other unfortunate side-effects.

To help you debug these problems, Fil will heuristically try to catch out-of-memory conditions, and dump a report if thinks your program is out of memory. It will then exit with exit code 53.

$ fil-profile run oom.py
...
=fil-profile= Wrote memory usage flamegraph to fil-result/2020-06-15T12:37:13.033/out-of-memory.svg

Fil uses three heuristics to determine if the process is close to running out of memory:

A failed allocation, indicating insufficient memory is available.
The operating system or memory-limited cgroup (e.g. a Docker container) only has 100MB of RAM available.
The process swap is larger than available memory, indicating heavy swapping by the process. In general you want to avoid swapping, and e.g. explicitly use mmap() if you expect to be using disk as a backfill for memory.

For a more detailed example of out-of-memory detection with Fil, see this article on debugging out-of-memory crashes.

Disabling the out-of-memory detection

Sometimes the out-of-memory detection heuristic will kick in too soon, shutting down the program even though in practice it could finish running. You can disable the heuristic by doing:

fil-profile --disable-oom-detection run yourprogram.py

Debugging memory leaks with Fil

Is your program suffering from a memory leak? You can use Fil to debug it.

Fil works by reporting the moment in your process lifetime where memory is highest. If your program has a memory leak, eventually the highest memory usage point is always the present, as leaked memory accumulates.

If for example your Python web application is leaking memory, you can:

Start it under Fil.
Generate lots of traffic that causes memory leaks.
When enough memory has leaked that it’s noticeable, cleanly kill the process (e.g. Ctrl-C).

Fil will then dump a report that will help pinpoint the leaking code.

For a more in-depth tutorial, read this article on debugging Python server memory leaks with Fil.

Disabling browser pop-up reports

By default, Fil will open the result of a profiling run in a browser.

As of version 2021.04.2, you can disable this by using the --no-browser option (see fil-profile --help for details). You will want to view the SVG report in a browser, since they rely heavily on JavaScript.

If you want to serve the report files from a static directory using a web server, you can do:

$ cd fil-result/
$ python -m http.server

Reference

Learn about: