The security scanner that cried wolf

by
Last updated 01 Oct 2021, originally created 03 Mar 2021

If you run a security scanner on your Docker image, you might be in for a shock: often you’ll be warned of dozens of security vulnerabilities, even on the most up-to-date image. After the third or fourth time you get this result, you’ll start tuning the security scanner out.

Eventually, you won’t pay attention to the security scanner at all—and you might end up missing a real security vulnerability that slipped through.

This is not your fault: the problem is the way many security scanners report their results. So let’s see what they output, why it’s problematic, and how to get more useful security scanner results.

Demonstrating the problem

Let’s say I create the following Dockerfile, using the latest stable release of Debian at the time of writing:

FROM debian:buster
RUN apt-get update && apt-get -y upgrade

I then rebuild it from scratch, to ensure I have actually updated everything:

$ docker build --no-cache --pull -t uptodate-debian .

And finally I push it to a well-known Docker image registry service. I won’t mention it by name because this isn’t about any specific service, it’s about a general problem.

The image registry performs a security scan, and it comes back saying there are 62 different security vulnerabilities in this image. One of them is high severity!

This is terrible!

Except… this image has every security update put out by Debian. It’s the stable release, Debian is a well-respected Linux distribution that puts out timely security updates—what’s going on?

If everything has security vulnerabilities, nothing has security vulnerabilities

As we’ve seen, pretty much every single image based on Debian Stable is going to be insecure. If that’s your default base image (as it is for most official Docker images that aren’t Alpine-based), that means all your images are going to be marked as insecure.

Which means you’re going to start ignoring these results; not a good outcome!

Alternatively, you might want to try to fix these vulnerabilities—but how, exactly? In this example I’ve made sure to install all updates, what else can you do?

Why scanners cry wolf

The problem with the security scanner results is that it’s listing CVEs that are never going to be fixed in this release of Debian Stable. In fact, some of them will never get fixed, in this or any future release. This is due to a number of reasons:

  1. It’s not actually a security vulnerability.
  2. It’s not something that can be fixed within the boundaries of a stable long-term-support release.
  3. It’s so trivial that there’s no point in fixing it.
  4. The upstream maintainers have decided not to release a fix.

Luckily, many of these vulnerabilities are also irrelevant to most or all container runtime environments.

To give you a sense of what I’m talking about, let’s look at a few of the vulnerabilities on the list.

CVE-2017-8804: package glibc, severity HIGH

Sounds bad, doesn’t it? But notice it’s from 2017—why hasn’t it been closed since?

Let’s read the description:

The xdr_bytes and xdr_string functions in the GNU C Library (aka glibc or libc6) 2.25 mishandle failures of buffer deserialization, which allows remote attackers to cause a denial of service (virtual memory allocation, or memory consumption if an overcommit setting is not used) via a crafted UDP packet to port 111, a related issue to CVE-2017-8779.

First, unless you’re running an NFS server or client inside your container—which seems highly unlikely—this isn’t something you need to worry about.

Second, if we follow the links we will eventually discover that the maintainers dispute this is a glibc issue, and consider it an application vulnerability. Their feeling is that whoever calls these APIs needs to do it in the appropriate way to make it secure. So seems like this is never going to get fixed in glibc.

CVE-2016-2781: package coreutils, severity LOW

Again, this is from 2016, quite a while ago. Here’s the description:

chroot in GNU coreutils, when used with –userspec, allows local users to escape to the parent session via a crafted TIOCSTI ioctl call, which pushes characters to the terminal’s input buffer.

First, you’re unlikely to be relying on the chroot command-line tool for security inside a container.

Second, it seems the maintainers aren’t going to fix this; there’s a note in the Debian security tracker saying:

Restricting ioctl on the kernel side seems the better approach, but rejected by Linux upstream. Fixing this issue via setsid() would introduce regressions.

Looking further, it sounds like the developers have added a new option that makes this tool more secure, but default behavior isn’t going to change. There isn’t much Debian can do about this without breaking compatibility.

CVE-2018-12886: package gcc-8, severity UNKNOWN

Here’s the description:

stack_protect_prologue in cfgexpand.c and stack_protect_epilogue in function.c in GNU Compiler Collection (GCC) 4.1 through 8 (under certain circumstances) generate instruction sequences when targeting ARM targets that spill the address of the stack protector guard, which allows an attacker to bypass the protection of -fstack-protector, -fstack-protector-all, -fstack-protector-strong, and -fstack-protector-explicit against stack overflow by controlling what the stack canary is compared against.

Basically, your C code will be more vulnerable to stack overflow attacks, if you’re running on ARM. Unless you’re using Graviton instances on AWS, chances are you’re on x86_64, and this CVE is irrelevant.

From Debian’s perspective, they’re not going to change to a major new version of GCC in a stable release. And while they will backport security fixes when possible, in this case they say that’s it’s “too intrusive to backport.”

If you do need this fix, you can’t get it within the confines of the Debian Buster stable release; it will only be available when a new Debian Stable ships with a newer major gcc release.

Comparing to RedHat

So far we’ve been looking at Debian; how does RedHat compare?

Let’s create a Docker image based on their Universal Base Image, with the version matching RHEL 8, and install updates:

FROM registry.access.redhat.com/ubi8/ubi
RUN dnf -y update

If we run the resulting image through the same security scanner—there are no security vulnerabilities!

Does that mean RedHat is more secure than Debian? Not really.

Looking at the first two CVEs we considered above, RedHat came to the same decisions as Debian: both issues were closed as WONTFIX (1, 2).

So why does the Debian-based image have so many security vulnerabilities listed, and the RedHat one does not? It seems to be some sort of reporting or data issue. For security issues that are in the WONTFIX category, the vulnerability databases used by the security scanner have the security issues marked as closed for RedHat, but not for Debian. It’s unclear to me where in the chain of communication and interpretation this issue happens.

But while this is annoying, it doesn’t really tell us anything about the security of Debian vs. RedHat. It just means we get more useful security scanner output for RedHat.

Making security scanners useful

Let’s take a step back and think about why we want a security vulnerability in the first place.

Unless you have the organizational resources to create custom security patches for 3rd-party software, which is unlikely, you are going to be relying on security updates from your Linux distribution of choice. Best practices for security include installing these updates as part of your Docker image build.

So the point of the security scanner is to catch cases where you’re forgotten to do that, or the update process failed (for example, due to Docker build caching).

In other words, what you really want to know is whether you’ve failed to install security updates.

Luckily, many security vulnerability scanners have an option for this. The command-line Trivy security scanner has an option called --ignore-unfixed. The registry-based security scanner I used in this article has a checkmark to “Only show fixable”.

So make sure to always use these options, and your security vulernability scanner output will become far more useful.

Sidenote: If you happen to work on a security scanner, please, make sure your scanner has a “show only fixable” option, and make sure it’s on by default.

Security updates are not enough

Security scanners are useful for detecting vulnerabilities not just in your system packages, but also in your Python dependencies, and even your own code. And since we’re talking about Docker, do make sure that Docker build caching doesn’t break your security updates.

Looking at the bigger picture, one thing we’ve learned is that many security vulnerabilities never get fixed. Most are trivial and unlikely, it’s true, but maybe one of them happens to apply to your particular setup. And chances are there are many more serious vulnerabilities that haven’t been found yet.

So you should to take further steps to ensure your Docker image is secure. Two examples: