Ayende @ Rahien

Hi!
My name is Oren Eini
Founder of Hibernating Rhinos LTD and RavenDB.
You can reach me by phone or email:

ayende@ayende.com

+972 52-548-6969

, @ Q c

Posts: 5,971 | Comments: 44,508

filter by tags archive

Dogfooding is CRITICAL, the story of a bug


We use RaveDB to handle our entire internal infrastructure. That has several reasons, to start with, RavenDB is a joy to work with, so it cuts down on the dev time for new internal features. More importantly, however, it let us try RavenDB in real world conditions. That seems obvious, because as much as you try, you can never really simulate real production environments.

But even more important is the concept of production data. The important thing about production data is that it is dirty. You don’t get data that is all nice and simple and all the same as you have when you generate the data, or when you are creating unit and system tests.

In this case, we had an interesting problem. We had a map reduce index similar to this one:

   1: public class UsersByCountry : AbstractIndexCreationTask<User, UsersByCountry.Result>
   2: {
   3:     public class Result
   4:     {
   5:         public string Country { get; set; }
   6:         public int Count { get; set; }
   7:     }
   8:  
   9:     public UsersByCountry()
  10:     {
  11:         Map = users =>
  12:               from user in users
  13:               select new {user.Country, Count = 1};
  14:         Reduce = results =>
  15:                  from result in results
  16:                  group result by result.Country
  17:                  into g
  18:                  select new
  19:                  {
  20:                      Country = g.Key,
  21:                      Count = g.Sum(x => x.Count)
  22:                  };
  23:     }
  24: }

This is a pretty standard thing to have, but I noticed that we had a difference from the 1.0 results in our production data. Investigating further, it appeared that this was the root issue:

   1: using (var session = store.OpenSession())
   2: {
   3:     session.Store(new User { Country = "Israel" });
   4:     session.Store(new User { Country = "ISRAEL" });
   5:     session.Store(new User { Country = "israel" });
   6:     session.SaveChanges();
   7: }

With RavenDB 1.0, due to fairly esoteric implementation details, this would generate the following values:

  • {"Country": "Israel", "Count": 1}
  • {"Country": "ISRAEL", "Count": 1}
  • {"Country": "israel", "Count": 1}

This matches what you’ll get using Linq to Objects, and that was fine by me. I could see an argument for doing the reduce using case insensitive reduce, but then you have the argument about which or the representation is the one you should use, etc.

In the version that I tested, I actually got:

  • {"Country": "Israel", "Count": 3}
  • {"Country": "ISRAEL", "Count": 3}
  • {"Country": "israel", "Count": 3}

Now, that was wrong. Very wrong.

As it turned out, once I managed to recover from the palpitations that this issue gave me, the actual reason was pretty easy to figure out. Some of our code was case sensitive, some of our code was not. That meant that under this condition, we would feed the map/reduce engine with duplicate entries, per the number of various casing combinations that we had.

Spooky bug, but once we narrowed down what the actual problem was, very easy to resolve.


Comments

Ruben Bartelink

Your mention of RaveDb gives me an idea for GabberDd :P

Dmitry

RaveDB actually sounds cooler than RavenDB. :)

peter

Out of tens of databases we have in our business, 3-4 of the unix-based ones were set up case-sensitive. This is forever tripping us up. What would be a scenario where you would want to differentiate between values based on case?

Ayende Rahien

Peter, What is the "master" value there?

pete w

very surprised this kind of bug wasnt found in production systems to date!

Ayende Rahien

Pete, It was introduced in the optimizations for 2.0

Comment preview

Comments have been closed on this topic.

FUTURE POSTS

  1. Paying the rent online - 2 days from now

There are posts all the way to Aug 03, 2015

RECENT SERIES

  1. Production postmortem (5):
    29 Jul 2015 - The evil licensing code
  2. Career planning (6):
    24 Jul 2015 - The immortal choices aren't
  3. API Design (7):
    20 Jul 2015 - We’ll let the users sort it out
  4. What is new in RavenDB 3.5 (3):
    15 Jul 2015 - Exploring data in the dark
  5. The RavenDB Comic Strip (3):
    28 May 2015 - Part III – High availability & sleeping soundly
View all series

Syndication

Main feed Feed Stats
Comments feed   Comments Feed Stats