Production postmortemThe case of the Unicode Poo

time to read 4 min | 635 words

We got an error report from a customer about migration issue from 2.5 to 3.0. A particular document appear to have been corrupted, and caused issues.

We have an explicit endpoint to expose the database raw bytes to the client, so we can troubleshoot exactly those kind of errors. For fun, this is a compressed database, so the error was hiding beneath two level of indirection, but that is beside the point.

When looking at the raw document’s byte, we saw:

image

Which was… suspicious. It took a while to track down, but the end result was that this error would occur when you have:

  • A large string (over 1KB), and it is the first large string in the document.
  • At the 1023 position of the string (bytewise), you have a multi byte and multiple character value.

In those cases, we wouldn’t be able to read the document.

The underlying reason was an optimization we made in 3.0 to reduce buffer allocations during deserialization of documents. In order to properly handle that, we used an Encoding Decoder directly, without any intermediate buffers. This works great, except in this scenario, and the way JSON.Net calls us.

When JSON.Net find a large string, it will repeatedly read characters from the stream until it reached the end of the stream, and only then it will process it. If the string size is more than the buffer size, it will increase the buffer.

Let us imagine the following string:

ABC

When we serialize it, it looks like this:

var bytes = new byte[] { 65, 66, 67, 0xF0, 0x9F, 0x92, 0xA9 };

And let us say that we want to read that in a buffer of 4 characters. We’ll use it like so:

 int bytesPos = 0;
 int charsPos = 0;
 var chars = new char[4];
 while (bytesPos < bytes.Length) // read whole buffer
 {
    while (charsPos < chars.Length) // process chars in chunks
     {
         int bytesUsed;
         int charsUsed;
         bool completed;
         decoder.Convert(bytes, bytesPos, bytes.Length - bytesPos, chars, charsPos, chars.Length - charsPos, false,
             out bytesUsed, out charsUsed, out completed);
         bytesPos += bytesUsed;
         charsPos += charsUsed;
     }
     Console.WriteLine(new string(chars));
 }

On the first call, the Convert will convert the first three bytes into three characters, and stop. The JSON.Net code will then ask it to fill to the end of the buffer (simulated by the inner loop), but at that point, the Convert method will throw, because it has just one character available in the buffer to write to, but it can’t write that character.

Why is that? Look at the poo string above. How many character does it take?

If you answered four, you are correct visually, and wrong in buffer sense. This string actually takes 5 characters to represent. As I mentioned, in order to hit this error, we have to had a particular set of things align just right (or wrong). Even a single space difference would align things so no multi byte character would span the 1KB boundary.

The solution, by the way, was to drop the optimization, sadly, we’ll revisit this at a later time, probably, but now we’ll have a way to confirm that this scenario is also covered.

More posts in "Production postmortem" series:

  1. (07 Aug 2017) 30% boost with a single line change
  2. (04 Aug 2017) The case of 99.99% percentile
  3. (02 Aug 2017) The lightly loaded trashing server
  4. (23 Aug 2016) The insidious cost of managed memory
  5. (05 Feb 2016) A null reference in our abstraction
  6. (27 Jan 2016) The Razor Suicide
  7. (13 Nov 2015) The case of the “it is slow on that machine (only)”
  8. (21 Oct 2015) The case of the slow index rebuild
  9. (22 Sep 2015) The case of the Unicode Poo
  10. (03 Sep 2015) The industry at large
  11. (01 Sep 2015) The case of the lying configuration file
  12. (31 Aug 2015) The case of the memory eater and high load
  13. (14 Aug 2015) The case of the man in the middle
  14. (05 Aug 2015) Reading the errors
  15. (29 Jul 2015) The evil licensing code
  16. (23 Jul 2015) The case of the native memory leak
  17. (16 Jul 2015) The case of the intransigent new database
  18. (13 Jul 2015) The case of the hung over server
  19. (09 Jul 2015) The case of the infected cluster