Ayende @ Rahien

It's a girl

The next BIG Thing in RavenDB 1.2

Is something that you probably wouldn’t even notice, to be perfectly honest. We are going to work on our map/reduce implementation.

This is freakishly complex, because we need to do updatable, persistent map/reduce. It got so complex that I decided that I can’t really implement this on my own in the RavenDB solution and moved to spiking the solution in isolation.

If you care, you can look at this here. There would be additional optimizations to worry about in RavenDB, but it is pretty much all there, in less than 400 lines of code.

I couldn’t find anything out there which was nearly as useful. Most of the map/reduce implementations are about distributing the work load and scheduling it. None of them really deal with the notion of updatable map/reduce results.

Note that the Storage layer there is both only there for the sole purpose of actually showing we can persist and restart from any point and also has critical behavior in its behavior (for example, scheduling).

I’ll probably do a set of posts about this, but for now, here is the source, have fun poking at it: https://github.com/ayende/updatable-persistent-map-reduce

Tags:

Posted By: Ayende Rahien

Published at

Originally posted at

Comments

Brian
08/10/2012 03:00 PM by
Brian

Really?

static void Main() { foreach (var directory in Directory.GetDirectories(".")) { try { Directory.Delete(directory, true); } catch (Exception) { } <snip...> }

Nicolas
08/10/2012 06:02 PM by
Nicolas

Take a look at this: https://github.com/nathanmarz/storm/

Maybe is not what you are looking for, or in the same language for that matter, but maybe it can give you some ideas... Or maybe not!

Daniel Lang
08/10/2012 09:22 PM by
Daniel Lang

One shouldn't run this exe from C:/ with admin privileges... lol

Ayende Rahien
08/11/2012 04:29 PM by
Ayende Rahien

Brian, Yes... ? This is there to make sure that we clear old results from the previous run.

Starfish
08/13/2012 11:40 AM by
Starfish

I trust there are no "breaking" changes?

Ayende Rahien
08/13/2012 11:44 AM by
Ayende Rahien

Starfish, Not outward facing, not from this.

Brian
08/13/2012 12:59 PM by
Brian

Oren, then might I suggest you at least put some sanity check around that block. Even though this is not production code, that's just leaving a loaded gun lying around (it's irresponsible coding and if you were reviewing any code that did this I'll bet you'd have blasted the author, too).

Ayende Rahien
08/13/2012 01:01 PM by
Ayende Rahien

Brian, That code is there to make my life easier. This project is there as a POC. I am not going to worry about it.

Matt Warren
08/17/2012 04:51 PM by
Matt Warren

"I couldn’t find anything out there which was nearly as useful. Most of the map/reduce implementations are about distributing the work load and scheduling it. None of them really deal with the notion of updatable map/reduce results."

You might want to take a look at Percolator, it's Google enhancement to Map/Reduce to solve this exact issue. I.e. so it doesn't have to re-create it's index from scratch every 2 weeks, it can do incremental updates.

See http://www.theregister.co.uk/2010/09/24/googlepercolator/ and http://static.googleusercontent.com/externalcontent/untrusted_dlcp/research.google.com/en//pubs/archive/36726.pdf

It might have some ideas that could be used in RavenDB

Ayende Rahien
08/18/2012 09:13 AM by
Ayende Rahien

Matt, I was mostly interested in going over source code, to see how the split & merge the work

Comments have been closed on this topic.