Ayende @ Rahien

My name is Oren Eini
Founder of Hibernating Rhinos LTD and RavenDB.
You can reach me by phone or email:


+972 52-548-6969

, @ Q c

Posts: 6,128 | Comments: 45,550

filter by tags archive

Dogfooding is CRITICAL, the story of a bug

time to read 11 min | 2184 words

We use RaveDB to handle our entire internal infrastructure. That has several reasons, to start with, RavenDB is a joy to work with, so it cuts down on the dev time for new internal features. More importantly, however, it let us try RavenDB in real world conditions. That seems obvious, because as much as you try, you can never really simulate real production environments.

But even more important is the concept of production data. The important thing about production data is that it is dirty. You don’t get data that is all nice and simple and all the same as you have when you generate the data, or when you are creating unit and system tests.

In this case, we had an interesting problem. We had a map reduce index similar to this one:

   1: public class UsersByCountry : AbstractIndexCreationTask<User, UsersByCountry.Result>
   2: {
   3:     public class Result
   4:     {
   5:         public string Country { get; set; }
   6:         public int Count { get; set; }
   7:     }
   9:     public UsersByCountry()
  10:     {
  11:         Map = users =>
  12:               from user in users
  13:               select new {user.Country, Count = 1};
  14:         Reduce = results =>
  15:                  from result in results
  16:                  group result by result.Country
  17:                  into g
  18:                  select new
  19:                  {
  20:                      Country = g.Key,
  21:                      Count = g.Sum(x => x.Count)
  22:                  };
  23:     }
  24: }

This is a pretty standard thing to have, but I noticed that we had a difference from the 1.0 results in our production data. Investigating further, it appeared that this was the root issue:

   1: using (var session = store.OpenSession())
   2: {
   3:     session.Store(new User { Country = "Israel" });
   4:     session.Store(new User { Country = "ISRAEL" });
   5:     session.Store(new User { Country = "israel" });
   6:     session.SaveChanges();
   7: }

With RavenDB 1.0, due to fairly esoteric implementation details, this would generate the following values:

  • {"Country": "Israel", "Count": 1}
  • {"Country": "ISRAEL", "Count": 1}
  • {"Country": "israel", "Count": 1}

This matches what you’ll get using Linq to Objects, and that was fine by me. I could see an argument for doing the reduce using case insensitive reduce, but then you have the argument about which or the representation is the one you should use, etc.

In the version that I tested, I actually got:

  • {"Country": "Israel", "Count": 3}
  • {"Country": "ISRAEL", "Count": 3}
  • {"Country": "israel", "Count": 3}

Now, that was wrong. Very wrong.

As it turned out, once I managed to recover from the palpitations that this issue gave me, the actual reason was pretty easy to figure out. Some of our code was case sensitive, some of our code was not. That meant that under this condition, we would feed the map/reduce engine with duplicate entries, per the number of various casing combinations that we had.

Spooky bug, but once we narrowed down what the actual problem was, very easy to resolve.


Ruben Bartelink

Your mention of RaveDb gives me an idea for GabberDd :P


RaveDB actually sounds cooler than RavenDB. :)


Out of tens of databases we have in our business, 3-4 of the unix-based ones were set up case-sensitive. This is forever tripping us up. What would be a scenario where you would want to differentiate between values based on case?

Ayende Rahien

Peter, What is the "master" value there?

pete w

very surprised this kind of bug wasnt found in production systems to date!

Ayende Rahien

Pete, It was introduced in the optimizations for 2.0

Comment preview

Comments have been closed on this topic.


  1. The worker pattern - 2 days from now

There are posts all the way to May 30, 2016


  1. The design of RavenDB 4.0 (14):
    26 May 2016 - The client side
  2. RavenDB 3.5 whirl wind tour (14):
    25 May 2016 - Got anything to declare, ya smuggler?
  3. Tasks for the new comer (2):
    15 Apr 2016 - Quartz.NET with RavenDB
  4. Code through the looking glass (5):
    18 Mar 2016 - And a linear search to rule them
  5. Find the bug (8):
    29 Feb 2016 - When you can't rely on your own identity
View all series


Main feed Feed Stats
Comments feed   Comments Feed Stats