Ayende @ Rahien

Hi!
My name is Oren Eini
Founder of Hibernating Rhinos LTD and RavenDB.
You can reach me by phone or email:

ayende@ayende.com

+972 52-548-6969

, @ Q c

Posts: 08 | Comments: 18

filter by tags archive

RavenDB Map/Reduce optimizations

time to read 2 min | 216 words

So I was diagnosing a customer problem, which required me to write the following:

image

This is working on a data set of about half a million records.

I took a peek at the stats and I saw this:

image

You can ignore everything before 03:23, this is the previous index run. I reset it to make sure that I have a clean test.

What you can see is that we start out with a mapping & reducing values. And you can see that initially this is quite expensive. But very quickly we recognize that we are reducing a single value, and we switch strategies to a more efficient method, and we suddenly have very little cost involved in here. In fact, you can see that the entire process took about 3 minutes from start to finish, and very quickly we got to the point where are bottle neck was actually the maps pushing data our way.

That is pretty cool.


Comments

Kyle Szklenski

I was looking at the graph first and thought, "Holy crap, that's a lot of work." Then I saw that it was done in only a second, and realized you weren't actually pointing out a problem, but some kind of trickery you do. Then I read the post. :) Good work, team.

Andreas Kroll

Awesome...

Sometimes you remind me of Hanibal Smith from the A-Team:

 "I love it, when a plan works..."

It's somehow similar to your trademark "it just works"

Comment preview

Comments have been closed on this topic.

FUTURE POSTS

  1. Concurrent max value - 9 hours from now
  2. Production postmortem: The case of the memory eater and high load - 3 days from now
  3. Production postmortem: The case of the lying configuration file - 4 days from now
  4. Production postmortem: The industry at large - 5 days from now
  5. The insidious cost of allocations - 6 days from now

And 5 more posts are pending...

There are posts all the way to Sep 10, 2015

RECENT SERIES

  1. Find the bug (5):
    20 Apr 2011 - Why do I get a Null Reference Exception?
  2. Production postmortem (10):
    14 Aug 2015 - The case of the man in the middle
  3. What is new in RavenDB 3.5 (7):
    12 Aug 2015 - Monitoring support
  4. Career planning (6):
    24 Jul 2015 - The immortal choices aren't
View all series

Syndication

Main feed Feed Stats
Comments feed   Comments Feed Stats