Yesterday I presented a bug that killed the process in a particularly rude manner. This is a recursive function that guards against stack overflows using RuntimeHelpers.EnsureSufficientExecutionStack().
Because of how this function kills the process, it took some time to figure out what is going on. There was no StackOverflowException, just an exit code. Here is the relevant code:
This looks okay, we optimize for zero allocations on the common path (less than 2K items), but also handle the big one.
The problem is that our math is wrong. More specifically, take a look at this line:
var sizeInBytes = o.Count / (sizeof(byte) * 8) + o.Count % (sizeof(byte) * 8) == 0 ? 0 : 1;
Let’s assume that your count is 10, what do you think the value of this is going to be?
Well, it looks like this should give us 2, right?
10 / 8 + 10%8 == 0 ? 0 :1
The problem is in the operator precedence. I read this as:
(10 / 8) + (10 % 8 == 0 ? 0 : 1)
And the C# compiler read it as:
(10 / 8 + 10 % 8) == 0 ? 0 : 1
In other words, *#@*!*@!.
The end result is that we overwrite past our allocated stack. Usually that doesn’t do anything bad, since there is enough stack space. But sometimes, if the stack is aligned just right, we cross into the stack guard page and kill the process.
Opps, that was not expected.