Ayende @ Rahien

My name is Oren Eini
Founder of Hibernating Rhinos LTD and RavenDB.
You can reach me by phone or email:


+972 52-548-6969

, @ Q c

Posts: 6,026 | Comments: 44,844

filter by tags archive

The smallest bugs, the biggest problems – Part II

time to read 2 min | 237 words

Originally posted at 11/22/2010

In a previous post, I talked about how I found the following (really nasty) bug in RavenDB’s managed storage (which is still considered unstable, btw):

When deleting documents in a database that contains more than 2 documents, and  the document(s) deleted are deleted in a certain order, RavenDB would go into 100% CPU. The server would still function, but it would always think that it had work to do, even if it didn’t have any.

Now, I want to talk about the actual bug.


What I did wrong here is to reuse the removed and value parameters in the second call to TryRemove. That call is internal, and is only needed to properly balance the tree, but what it ended up doing is always return the removed/value from the right side of the tree.

Compounding the problem is that I only actually used the TryRemove value in a single location, and even then, it is a mistake. Take a look:


That meant that I actually looked for the problem in the secondary indexes for a while, before realizing that the actual problem was elsewhere.


Rob Kent

When I worked in the mainframe world, one of the support guys, whose name was Ken Russell, came up with what we called 'Russell's Law':

'The length of time it takes to find a bug is inversely proportional to the size of the fix'.

A misplaced byte could take days to find and minute to fix, whereas an obvious bug required a page of code to patch.


I assume it's nothing more than a typo... but where does theValue come from?


Ade - A private field?


out arguments shit.. That's why i prefer to create a special class to return more values.

Wesner Moise

Since this is an functional algorithm, you should use a different strategy for detecting actual removal.

1) Check before and after counts


2) Compare to see if the new tree is identical to the previous one after the operation

Comment preview

Comments have been closed on this topic.


No future posts left, oh my!


  1. Technical observations from my wife (3):
    13 Nov 2015 - Production issues
  2. Production postmortem (13):
    13 Nov 2015 - The case of the “it is slow on that machine (only)”
  3. Speaking (5):
    09 Nov 2015 - Community talk in Kiev, Ukraine–What does it take to be a good developer
  4. Find the bug (5):
    11 Sep 2015 - The concurrent memory buster
  5. Buffer allocation strategies (3):
    09 Sep 2015 - Bad usage patterns
View all series


Main feed Feed Stats
Comments feed   Comments Feed Stats