Production postmortemThe network latency and the I/O spikes

time to read 4 min | 669 words

A customer called us about an elevated I/O load in their system after an upgrade to RavenDB 5.3 from RavenDB 4.2. We looked into that, and we saw a small (but very noticeable) rise that we just could not explain. Those sorts of issues are tough to crack, because there isn’t an error or a smoking gun to get you started.

Instead, we just saw a higher average I/O rate, but what is the reason for that? Maybe it is a seasonal change for the customer, with a higher load during the springtime? Or maybe it is related to a new index that was deployed?

We looked, but there hasn’t been anything that should cause higher I/O stress for the system that we could see. So we started diving deeper and deeper into the metrics. On Linux, you can check what files are being read or written to (and all of those that we could see represented reasonable values for their load, there wasn’t anything that wasn’t expected). You can also pull the I/O stats by thread, and we could see that the cluster threads were quite busy in terms of I/O, but that is a big cluster, with plenty of databases and cluster operations to manage, it seems reasonable.

What is going on? I just checked, and the timeline for this investigation is about four weeks, we tried a lot of things to figure it out. But we couldn’t find a smoking gun.

Separately, we got a few bug reports from the field about a cluster issue, sometimes the cluster connection between nodes would break for no reason. The connectivity was good, so there was no reason for the break. This is a transient (and expected) error, which RavenDB will gracefully recover from. But it was a new behavior, so we looked into that.

It turns out that during some refactoring, we moved a piece of code in such a way that under certain conditions, it would read too much from the buffer, but not consume all of it. Basically, this issue came back in some cases. In order to trigger this problem, we had to have a very specific network configuration with exact latencies compared to the CPU load on the server. When that behavior was triggered, we would discard some part of the message from the other side. In some cases, that just meant that we skipped an update (in a stream of them), no big deal, we’ll get the next one successfully. But depending on the size of the cluster in question and the latencies involved, we may get corrupted data (since we are missing the data). We properly detect and abort the connection in this case.

It turns out that when such a thing happens, RavenDB considers the other side to have failed, and the cluster takes the appropriate action to compensate. That means that it will re-assign the tasks across the cluster. A few seconds later, the connection would be resumed, the cluster would realize that the node is “up” again and move the tasks back to the node.

Those tasks include things like subscriptions, ETL processes, external replication, etc.

In other words, under a specific set of conditions, we’ll have a lot of jitters, for lack of a better term in the cluster. Some of the nodes will be moved in and out of rehab (a status that means that they aren’t fully functional). That led, in turn, to a high churn of tasks (and each of those has its own I/O costs).

There are other factors here, naturally, such as higher CPU and memory, but I/O is where we are typically most constrained, so it showed up there mostly. The bug was fixed (and it is in the latest stable) and we have confirmation from the customer that this indeed resolved their issue.

It just goes to show how complex systems are. A bug occurs on node A when reading from the network under specific latencies conditions has cascaded to a higher resource utilization on node C. Butterfly effect indeed.

More posts in "Production postmortem" series:

  1. (15 Sep 2022) The missed indexing reference
  2. (05 Aug 2022) The allocating query
  3. (22 Jul 2022) Efficiency all the way to Out of Memory error
  4. (18 Jul 2022) Broken networks and compressed streams
  5. (13 Jul 2022) Your math is wrong, recursion doesn’t work this way
  6. (12 Jul 2022) The data corruption in the node.js stack
  7. (11 Jul 2022) Out of memory on a clear sky
  8. (29 Apr 2022) Deduplicating replication speed
  9. (25 Apr 2022) The network latency and the I/O spikes
  10. (22 Apr 2022) The encrypted database that was too big to replicate
  11. (20 Apr 2022) Misleading security and other production snafus
  12. (03 Jan 2022) An error on the first act will lead to data corruption on the second act…
  13. (13 Dec 2021) The memory leak that only happened on Linux
  14. (17 Sep 2021) The Guinness record for page faults & high CPU
  15. (07 Jan 2021) The file system limitation
  16. (23 Mar 2020) high CPU when there is little work to be done
  17. (21 Feb 2020) The self signed certificate that couldn’t
  18. (31 Jan 2020) The slow slowdown of large systems
  19. (07 Jun 2019) Printer out of paper and the RavenDB hang
  20. (18 Feb 2019) This data corruption bug requires 3 simultaneous race conditions
  21. (25 Dec 2018) Handled errors and the curse of recursive error handling
  22. (23 Nov 2018) The ARM is killing me
  23. (22 Feb 2018) The unavailable Linux server
  24. (06 Dec 2017) data corruption, a view from INSIDE the sausage
  25. (01 Dec 2017) The random high CPU
  26. (07 Aug 2017) 30% boost with a single line change
  27. (04 Aug 2017) The case of 99.99% percentile
  28. (02 Aug 2017) The lightly loaded trashing server
  29. (23 Aug 2016) The insidious cost of managed memory
  30. (05 Feb 2016) A null reference in our abstraction
  31. (27 Jan 2016) The Razor Suicide
  32. (13 Nov 2015) The case of the “it is slow on that machine (only)”
  33. (21 Oct 2015) The case of the slow index rebuild
  34. (22 Sep 2015) The case of the Unicode Poo
  35. (03 Sep 2015) The industry at large
  36. (01 Sep 2015) The case of the lying configuration file
  37. (31 Aug 2015) The case of the memory eater and high load
  38. (14 Aug 2015) The case of the man in the middle
  39. (05 Aug 2015) Reading the errors
  40. (29 Jul 2015) The evil licensing code
  41. (23 Jul 2015) The case of the native memory leak
  42. (16 Jul 2015) The case of the intransigent new database
  43. (13 Jul 2015) The case of the hung over server
  44. (09 Jul 2015) The case of the infected cluster