Your Docker build needs a smoke test

Software needs automated tests to ensure changes don’t break: from unit test to end-to-end tests. And just like other software, your Docker image can also break.

The problem with a broken Docker image is that:

1. Unit tests won’t catch the problem.
2. End-to-end tests will catch it too late.

Let’s see why this is, and then talk about the solution: a smoke test for your Docker image.

Building a broken image

When you’re building a Docker image, the build process likely goes something like this:

1. Run unit tests on the code.
2. Build the Docker image.
3. Run end-to-end tests using the new Docker image.
4. Deploy to production.

If any step fails, the process stops. Failing to build the image will stop the process, because docker build will fail: COPYing a non-existent file, or RUNing a bad command.

But what if you successfully built an image, but that image is broken?

• Your Dockerfile’s CMD or ENTRYPOINT might be wrong, so when you start a container it can’t find the executable and crashes.
• The entrypoint script is found, but it’s buggy, so it crashes.
• Your image might listen on 127.0.0.1, so the server will start but you won’t be able to connect to it.
• Your web server starts, but can’t find the code, so every query responds with a 500 error.

Neither unit tests nor docker build will catch these problems. Unit tests won’t catch them because they’re packaging issues, and docker build won’t catch them because they only happen when you run the image, not during the build.

End-to-end tests are too late

If you don’t have end-to-end tests at all and you just deploy a Docker image as soon as it’s built, you’re liable to break production when you create a broken image.

But while end-to-end tests will catch broken Docker images, they also do so in a way that will waste a lot of developer time:

• Rather than immediately noticing the build fails, you will often only get told after all the end-to-end tests fail, which depending on your test suite can be quite a while.
• You won’t be told “the Docker image is broken”. Instead, you’ll have to dig through lots of logs to figure out what actually happened, and the symptom might well be an obscure timeout.

To avoid wasting time, you want to catch the problems with your Docker image earlier in the process, before the end-to-end tests.

The solution: a smoke test

A smoke test checks that the basic functionality of your code works: it won’t catch everything, but it will catch utter brokenness. And given end-to-end tests, that’s good enough for testing your Docker image build.

Let’s say you’re building a Docker image for a HTTP server. A basic smoke test would:

1. Start a container using the image.
2. Send a HTTP query to the server. Ideally the query would ensure some minimal amount of the code is imported and run, but it doesn’t have to test any of the business logic.
3. Check for an OK response.

For example:

import time
from subprocess import check_call
from urllib.request import urlopen

check_call(
    "docker run --rm --name=smk -p 8080:80 -d httpd".split()
)
# Wait for the server to start. A better implementation
# would poll in a loop:
time.sleep(5)
# Check if the server started (it'll throw an exception
# if not):
try:
    urlopen("http://localhost:8080").read()
finally:
    check_call("docker kill smk".split())

In all of the failure modes we mentioned above—crashing on startup, listening on the wrong port, not finding the code—this smoke test will catch the problem, fail, and stop the build process. And unlike end-to-end tests, you’ll know why the build process failed: something is wrong with your Docker packaging.

In more complex cases you might use Docker Compose to start up some dependent services, but in general your goal is the most minimal test that will demonstrate your image isn’t completely broken.

Once you’ve written a smoke test, your build process will look like this:

1. Run unit tests on the code.
2. Build the Docker image.
3. 🔥 Run your new smoke test. 🔥
4. Run end-to-end tests using the new Docker image.
5. Deploy to production.

Easy, fast, and very useful

Smoke tests won’t catch every problem, but they don’t need to: that’s why you have unit tests and end-to-end tests. What they will do is save you lots of time debugging mysterious end-to-end test failures.

And since smoke tests are easy to write and fast to run, there’s no reason not to implement one today.