Production postmortemThe case of the Unicode Poo

time to read 4 min | 635 words

We got an error report from a customer about migration issue from 2.5 to 3.0. A particular document appear to have been corrupted, and caused issues.

We have an explicit endpoint to expose the database raw bytes to the client, so we can troubleshoot exactly those kind of errors. For fun, this is a compressed database, so the error was hiding beneath two level of indirection, but that is beside the point.

When looking at the raw document’s byte, we saw:


Which was… suspicious. It took a while to track down, but the end result was that this error would occur when you have:

  • A large string (over 1KB), and it is the first large string in the document.
  • At the 1023 position of the string (bytewise), you have a multi byte and multiple character value.

In those cases, we wouldn’t be able to read the document.

The underlying reason was an optimization we made in 3.0 to reduce buffer allocations during deserialization of documents. In order to properly handle that, we used an Encoding Decoder directly, without any intermediate buffers. This works great, except in this scenario, and the way JSON.Net calls us.

When JSON.Net find a large string, it will repeatedly read characters from the stream until it reached the end of the stream, and only then it will process it. If the string size is more than the buffer size, it will increase the buffer.

Let us imagine the following string:


When we serialize it, it looks like this:

var bytes = new byte[] { 65, 66, 67, 0xF0, 0x9F, 0x92, 0xA9 };

And let us say that we want to read that in a buffer of 4 characters. We’ll use it like so:

 int bytesPos = 0;
 int charsPos = 0;
 var chars = new char[4];
 while (bytesPos < bytes.Length) // read whole buffer
    while (charsPos < chars.Length) // process chars in chunks
         int bytesUsed;
         int charsUsed;
         bool completed;
         decoder.Convert(bytes, bytesPos, bytes.Length - bytesPos, chars, charsPos, chars.Length - charsPos, false,
             out bytesUsed, out charsUsed, out completed);
         bytesPos += bytesUsed;
         charsPos += charsUsed;
     Console.WriteLine(new string(chars));

On the first call, the Convert will convert the first three bytes into three characters, and stop. The JSON.Net code will then ask it to fill to the end of the buffer (simulated by the inner loop), but at that point, the Convert method will throw, because it has just one character available in the buffer to write to, but it can’t write that character.

Why is that? Look at the poo string above. How many character does it take?

If you answered four, you are correct visually, and wrong in buffer sense. This string actually takes 5 characters to represent. As I mentioned, in order to hit this error, we have to had a particular set of things align just right (or wrong). Even a single space difference would align things so no multi byte character would span the 1KB boundary.

The solution, by the way, was to drop the optimization, sadly, we’ll revisit this at a later time, probably, but now we’ll have a way to confirm that this scenario is also covered.

More posts in "Production postmortem" series:

  1. (07 Jun 2019) Printer out of paper and the RavenDB hang
  2. (18 Feb 2019) This data corruption bug requires 3 simultaneous race conditions
  3. (25 Dec 2018) Handled errors and the curse of recursive error handling
  4. (23 Nov 2018) The ARM is killing me
  5. (22 Feb 2018) The unavailable Linux server
  6. (06 Dec 2017) data corruption, a view from INSIDE the sausage
  7. (01 Dec 2017) The random high CPU
  8. (07 Aug 2017) 30% boost with a single line change
  9. (04 Aug 2017) The case of 99.99% percentile
  10. (02 Aug 2017) The lightly loaded trashing server
  11. (23 Aug 2016) The insidious cost of managed memory
  12. (05 Feb 2016) A null reference in our abstraction
  13. (27 Jan 2016) The Razor Suicide
  14. (13 Nov 2015) The case of the “it is slow on that machine (only)”
  15. (21 Oct 2015) The case of the slow index rebuild
  16. (22 Sep 2015) The case of the Unicode Poo
  17. (03 Sep 2015) The industry at large
  18. (01 Sep 2015) The case of the lying configuration file
  19. (31 Aug 2015) The case of the memory eater and high load
  20. (14 Aug 2015) The case of the man in the middle
  21. (05 Aug 2015) Reading the errors
  22. (29 Jul 2015) The evil licensing code
  23. (23 Jul 2015) The case of the native memory leak
  24. (16 Jul 2015) The case of the intransigent new database
  25. (13 Jul 2015) The case of the hung over server
  26. (09 Jul 2015) The case of the infected cluster