Ayende @ Rahien

It's a girl

Dogfooding is CRITICAL, the story of a bug

We use RaveDB to handle our entire internal infrastructure. That has several reasons, to start with, RavenDB is a joy to work with, so it cuts down on the dev time for new internal features. More importantly, however, it let us try RavenDB in real world conditions. That seems obvious, because as much as you try, you can never really simulate real production environments.

But even more important is the concept of production data. The important thing about production data is that it is dirty. You don’t get data that is all nice and simple and all the same as you have when you generate the data, or when you are creating unit and system tests.

In this case, we had an interesting problem. We had a map reduce index similar to this one:

   1: public class UsersByCountry : AbstractIndexCreationTask<User, UsersByCountry.Result>
   2: {
   3:     public class Result
   4:     {
   5:         public string Country { get; set; }
   6:         public int Count { get; set; }
   7:     }
   8:  
   9:     public UsersByCountry()
  10:     {
  11:         Map = users =>
  12:               from user in users
  13:               select new {user.Country, Count = 1};
  14:         Reduce = results =>
  15:                  from result in results
  16:                  group result by result.Country
  17:                  into g
  18:                  select new
  19:                  {
  20:                      Country = g.Key,
  21:                      Count = g.Sum(x => x.Count)
  22:                  };
  23:     }
  24: }

This is a pretty standard thing to have, but I noticed that we had a difference from the 1.0 results in our production data. Investigating further, it appeared that this was the root issue:

   1: using (var session = store.OpenSession())
   2: {
   3:     session.Store(new User { Country = "Israel" });
   4:     session.Store(new User { Country = "ISRAEL" });
   5:     session.Store(new User { Country = "israel" });
   6:     session.SaveChanges();
   7: }

With RavenDB 1.0, due to fairly esoteric implementation details, this would generate the following values:

  • {"Country": "Israel", "Count": 1}
  • {"Country": "ISRAEL", "Count": 1}
  • {"Country": "israel", "Count": 1}

This matches what you’ll get using Linq to Objects, and that was fine by me. I could see an argument for doing the reduce using case insensitive reduce, but then you have the argument about which or the representation is the one you should use, etc.

In the version that I tested, I actually got:

  • {"Country": "Israel", "Count": 3}
  • {"Country": "ISRAEL", "Count": 3}
  • {"Country": "israel", "Count": 3}

Now, that was wrong. Very wrong.

As it turned out, once I managed to recover from the palpitations that this issue gave me, the actual reason was pretty easy to figure out. Some of our code was case sensitive, some of our code was not. That meant that under this condition, we would feed the map/reduce engine with duplicate entries, per the number of various casing combinations that we had.

Spooky bug, but once we narrowed down what the actual problem was, very easy to resolve.

Tags:

Posted By: Ayende Rahien

Published at

Originally posted at

Comments

Ruben Bartelink
11/27/2012 10:50 AM by
Ruben Bartelink

Your mention of RaveDb gives me an idea for GabberDd :P

Dmitry
11/27/2012 01:19 PM by
Dmitry

RaveDB actually sounds cooler than RavenDB. :)

peter
11/27/2012 05:55 PM by
peter

Out of tens of databases we have in our business, 3-4 of the unix-based ones were set up case-sensitive. This is forever tripping us up. What would be a scenario where you would want to differentiate between values based on case?

Ayende Rahien
11/27/2012 05:56 PM by
Ayende Rahien

Peter, What is the "master" value there?

pete w
11/28/2012 05:14 PM by
pete w

very surprised this kind of bug wasnt found in production systems to date!

Ayende Rahien
11/29/2012 09:16 AM by
Ayende Rahien

Pete, It was introduced in the optimizations for 2.0

Comments have been closed on this topic.