Dogfooding is CRITICAL, the story of a bug
We use RaveDB to handle our entire internal infrastructure. That has several reasons, to start with, RavenDB is a joy to work with, so it cuts down on the dev time for new internal features. More importantly, however, it let us try RavenDB in real world conditions. That seems obvious, because as much as you try, you can never really simulate real production environments.
But even more important is the concept of production data. The important thing about production data is that it is dirty. You don’t get data that is all nice and simple and all the same as you have when you generate the data, or when you are creating unit and system tests.
In this case, we had an interesting problem. We had a map reduce index similar to this one:
1: public class UsersByCountry : AbstractIndexCreationTask<User, UsersByCountry.Result>2: {
3: public class Result4: {
5: public string Country { get; set; }6: public int Count { get; set; }7: }
8:
9: public UsersByCountry()10: {
11: Map = users =>
12: from user in users13: select new {user.Country, Count = 1};14: Reduce = results =>
15: from result in results16: group result by result.Country
17: into g
18: select new19: {
20: Country = g.Key,
21: Count = g.Sum(x => x.Count)
22: };
23: }
24: }
This is a pretty standard thing to have, but I noticed that we had a difference from the 1.0 results in our production data. Investigating further, it appeared that this was the root issue:
1: using (var session = store.OpenSession())2: {
3: session.Store(new User { Country = "Israel" });4: session.Store(new User { Country = "ISRAEL" });5: session.Store(new User { Country = "israel" });6: session.SaveChanges();
7: }
With RavenDB 1.0, due to fairly esoteric implementation details, this would generate the following values:
- {"Country": "Israel", "Count": 1}
- {"Country": "ISRAEL", "Count": 1}
- {"Country": "israel", "Count": 1}
This matches what you’ll get using Linq to Objects, and that was fine by me. I could see an argument for doing the reduce using case insensitive reduce, but then you have the argument about which or the representation is the one you should use, etc.
In the version that I tested, I actually got:
- {"Country": "Israel", "Count": 3}
- {"Country": "ISRAEL", "Count": 3}
- {"Country": "israel", "Count": 3}
Now, that was wrong. Very wrong.
As it turned out, once I managed to recover from the palpitations that this issue gave me, the actual reason was pretty easy to figure out. Some of our code was case sensitive, some of our code was not. That meant that under this condition, we would feed the map/reduce engine with duplicate entries, per the number of various casing combinations that we had.
Spooky bug, but once we narrowed down what the actual problem was, very easy to resolve.
Comments
<Pendantico>Your mention of RaveDb gives me an idea for GabberDd :P</Pendantico>
RaveDB actually sounds cooler than RavenDB. :)
Out of tens of databases we have in our business, 3-4 of the unix-based ones were set up case-sensitive. This is forever tripping us up. What would be a scenario where you would want to differentiate between values based on case?
Peter, What is the "master" value there?
very surprised this kind of bug wasnt found in production systems to date!
Pete, It was introduced in the optimizations for 2.0
Comment preview