Error Absorption: When Static Analysis hides your bugs

Reading Time: 6 minutes

After Static Analysis has found a bug, it must make assumptions about how to continue the analysis. In this article, we learn about a common “filtering effect” and the pitfalls around it. At the end of this article, you will not only understand why bugs can hide other bugs, but also why that is not a bad thing, and how you can influence it to suit your taste.

Sometimes, Static Analysis can be confusing to us: It finds a bug, and in the next few lines of code, we observe weird effects like obvious flaws that are no longer reported as bugs. In some cases, this can be an annoying tool behavior. But in most cases, it is caused by the way Static Analysis works: Think of a buffer overflow – what happens after that? Can we even continue to analyze? The simple answer may be – no, we must stop analyzing there.

However, this answer falls short. As we have discussed before, Static Analysis can never be perfectly precise and must handle uncertainty. Therefore, Static Analysis needs a “way forward”, even when it detects a buffer overflow. And that can be confusing, even to experienced Static Analysis users like myself.

I am talking about “Error Absorption” – a situation where Static Analysis hides bugs from you. Let’s dive into this concept so that we can better understand our Static Analysis tool.

Run-Time behavior – What we observe in reality

Assume we have the following C code. What would happen at run-time:

int x = rand();
assert(x != 0);  //<<< this can fail
y = 12 / x;      //<<< this is safe

Here, x is assigned a random value, and the assertion might fail depending on how lucky we are. If we are unlucky and x becomes zero, the assertion evaluates to false, and the execution stops after the second line (I am ignoring the case where some madman turned off the assertions).

Now, the interesting part. The third line will create a division by zero if and only if x is zero. However, x cannot be zero, since the assertion in the previous line has “filtered out” this offending value. Consequently, the division is always safe.

The Static Analysis viewpoint

Now, let’s look from a Static Analysis perspective. What does the tool think? It turns out that reality and analysis can diverge, and that confuses people. However, it is actually pretty intuitive.

Example 1: When analysis and run-time agree

If we feed the above program into a sound static code analysis tool, the result will be exactly as we expect:

int x = rand();
assert(x != 0);  //<<< here is a bug
y = 12 / x;      //<<< here is no bug

• Static Analysis reports a bug on the assertion that depends on the value of x,
• but it does not report a bug on the division, since x cannot be zero here.

In other words, a sound static code analysis tool will exactly match the run-time behavior on this example. No surprise here.

Example 2: When analysis and run-time can diverge

A slightly different program can already lead to confusion, since Static Analysis and run-time behavior are no longer the same:

int array[3];
int x = rand();
y = array[x];  //<<< here is a bug
z = array[x];  //<<< here is no bug - but why??

In the third line, there is the risk of an out-of-bounds array access. The function rand returns a number between zero and very-big-number. If the number is larger than the length of array, it will be our unlucky day.

Run-Time behavior: For languages like C and C++, there might be a segmentation fault that stops execution after the third line. But there is no guarantee for that. The execution might also continue, since this is Undefined Behavior. We could reach line four, where the same bug happens again.

Static Code Analysis: Unlike the run-time behavior, static analysis cannot continue to analyze after a faulty behavior. Of course, it reports a bug on line three. The surprising part, however, is that it does not report a bug on line four.

Error Path Pruning – a necessary filter

Whenever Static Analysis finds a bug, it analyzes subsequent operations under the assumption that the bug never happened. This is called error path pruning, error absorption, or sometimes blocking state semantics, and part of the flow analysis. In our latest example, this means that after reporting the bug in line three, static analysis continues with the assumption that the value x is now in a valid range for the array access, namely in the range 0 through 2. This is like a “filter”, which results in the fourth line being judged as error-free.

The same happens for overflows, divisions by zero, and whatever else the static analysis tool is checking. This is necessary for Undefined Behavior because there is no way to model it in analysis. After all, it is undefined by definition. The program might leak your memory (hello, SSL Heartbleed), it might change execution flow, it might knock out all airports for a few days, or nothing might happen. In summary, a sound static analysis has no other choice than applying path pruning; everything else would be guesswork

Side note: Some compilers (especially C and C++) behave the same when optimizations are enabled. They will assume that these things never happen, and may even disable defensive coding was intended to avoid undefined behavior (scary but true, see here). This is an unfavorable application of error path pruning, of course, and one that happens without actually knowing whether there is an error.

Pros and cons of error path pruning

Besides being necessary, path pruning has a positive effect on our code review: The same bug is not reported several times (that is why gcc calls this behavior suppress followups). We only see the first error location, which is probably closest to the root cause. If we fix the bug, then subsequent buggy locations (which we don’t see) are also fixed.

And that’s exactly the downside: We don’t know what else is hiding, unless we have fixed the buggy locations. This means that after fixing bugs and re-analyzing the code, we may find new bugs, even more than before. In other words, bugs are hiding other bugs.

Does it make Static Analysis worse?

As we have seen, this cannot be answered generically. It depends on your personal goals:

• It’s worse if you want to see all potential bugs at once, and not uncover them with iterative bug fixing.
• It’s better because it produces fewer cascading warnings and points closer to the root cause.

And lastly, it is neither good nor bad. The first buggy location will anyway create some harm, and may lead to program termination. There is simply no way to continue analysing. Even if we could somehow “account for the unknown”, it would create a lot of False Positives. And nobody likes those except me.

Upcoming Article

Next time, we will look into how you can leverage path pruning to reduce False Positives.