Debugging ImportError and ModuleNotFoundErrors in your Docker image

Your code runs fine on your computer, but when you try to package it with Docker you keep getting ImportErrors or ModuleNotFoundError: Python can’t find your code.

There are multiple reasons why this can happen, some of them Python-specific, some of them Docker-specific. So let’s go through a step-by-step process to figuring out what the problem is, and how to fix it.

Step 1. Are you using the right Python interpreter?

It’s easy to mistakenly end up with multiple Python interpreters in your Docker image. When that happens, you might install your code with interpreter A, but try to run it with interpreter B—which then can’t find the code.

Here’s a somewhat contrived example of how this happens:

FROM python:3.8-slim-buster
RUN apt-get update && apt-get install -y python3 python3-pip
RUN /usr/bin/pip3 install flask
ENTRYPOINT ["python", "-c", "import flask"]

Note: Outside any specific best practice being demonstrated, the Dockerfiles in this article are not examples of best practices, since the added complexity would obscure the main point of the article.

Need to ship quickly, and don’t have time to figure out every detail on your own? Read the concise, action-oriented Python on Docker Production Handbook.

You now have two Python interpreters:

1. /usr/local/bin/python is the Python interpreter provided by the Docker image. This is what the ENTRYPOINT is running.
2. In /usr/bin/python3 is the Python interpreter installed via apt-get. This is what /usr/bin/pip3 will use.

Sometimes this happens less visibly, when you install a system package that depends on a python or python3 system package, or try to install a library by doing apt-get install python3-numpy.

Solution

This mostly happens if you’re (mis)using the official python base image. If you are:

1. Don’t manually install Python.
2. If you’re installing system packages, check if Python is one of the dependent system packages installed, and if so make sure you’re using the /usr/local/bin/ version.
3. Install Python libraries using pip, not apt-get.

Step 2. If relevant, are you successfully activating your virtualenv/conda env?

If you’re using an isolated environment like a virtualenv or Conda environment, you’re in a similar situation to the above: you have two different versions of Python, the system Python and the isolated environment. You want to make sure your code is installed and run from the same one.

Solution

Make sure you are correctly activating your virtualenv, or correctly activating your Conda environment.

Step 3. If it’s not pip installed, make sure the code location is correct

This is where you need to start understanding how Python decides where you can import code from.

To begin with, Python has a series of standard directories where it checks for imports. You can seem them by looking at sys.path:

$ docker run -it python:3.8-slim-buster
Python 3.8.3 (default, Jun  9 2020, 17:49:41) 
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.path
['', '/usr/local/lib/python38.zip', '/usr/local/lib/python3.8', '/usr/local/lib/python3.8/lib-dynload', '/usr/local/lib/python3.8/site-packages']

We’ll ignore that first item, the '', for now. The next few are basically just to give you access to the Python standard library. The last entry, ending with site-packages, is where pip will install packages.

So long as you’re installing your code with pip, you’re all good. It doesn’t matter what directory you’re in, you will be able to import it successfully.

$ docker run -it python:3.8-slim-buster bash
root@3c19e2314f01:/# pip install --quiet flask
root@3c19e2314f01:/# python -c "import flask; print('success')"
success
root@3c19e2314f01:/# cd /tmp
root@3c19e2314f01:/tmp# python -c "import flask; print('success')"
success

The problem usually arises when you’re not installing code with pip, but rather just copying it into some arbitrary directory.

That’s where the first entry in sys.path, the blank string '', comes into effect. What '' means is “add the directory where the initial script you ran is to sys.path.” Since that may be a little confusing, let’s look at two examples.

Let’s say we have two files, main.py and library.py; main.py will do an import library. If they are in the same directory, everything will work:

$ tree
.
└── code
    ├── library.py
    └── main.py

1 directory, 2 files
$ python code/main.py 
Successfully imported library.py
$ cd code/
$ python main.py 
Successfully imported library.py

If they are in different directories, the import will fail:

$ tree
.
├── code
│  └── library.py
└── main.py

1 directory, 2 files
$ python main.py 
Traceback (most recent call last):
  File "main.py", line 1, in <module>
    import library
ModuleNotFoundError: No module named 'library'
$ cd code/
$ python ../main.py 
Traceback (most recent call last):
  File "../main.py", line 1, in <module>
    import library
ModuleNotFoundError: No module named 'library'

Notice that your current working directory doesn’t matter. What matters is that the code you’re importing is in the same directory as the main script you’re running.

Solution

Either install all your code with pip install, or make sure it’s all in the same directory.

Step 4. python -m requires the code to be in the current working directory

If you’re using python -m yourpackage or python -m yourmodule, and you haven’t just pip installed everything, then you need to follow the same requirements as step 3: the imported code needs to be in the same directory as the main script.

In addition, your current directory (which you can set in your Dockerfile using WORKDIR), must be the same directory where your code sits.

$ tree
.
└── code
    ├── library.py
    └── main.py

1 directory, 2 files
$ python -m main
/usr/bin/python: No module named main
$ cd code/
$ python -m main
Successfully imported library.py

If your code is a package, you need to be in the directory containing the package:

$ tree
.
├── library.py
└── myapp
    ├── __init__.py
    └── __main__.py

1 directory, 3 files
$ python -m myapp
Successfully imported library.py
$ cd myapp/
$ python -m myapp
/usr/bin/python: No module named myapp

Solution

Make sure your current working directory is the same as the code:

FROM python:3.8-slim-buster
WORKDIR /code
COPY myapp .

The short version

To recap, here are the suggested actions to take to prevent ImportErrors:

1. Make sure you only have one version of Python installed.
2. If you’re using a virtualenv/conda env, make sure it’s correctly activated.
3. Either install all your code with pip, or:
   1. Make sure it’s all in the same directory.
   2. If you’re additionally using python -m, make sure your working directory is the same as the directory the code resides in.

Want to learn more debugging techniques for common Docker packaging problems? There’s a whole chapter covering 13 different debugging techniques in Just Enough Docker Packaging, my introductory book for Python programmers.