Ayende @ Rahien

Hi!
My name is Oren Eini
Founder of Hibernating Rhinos LTD and RavenDB.
You can reach me by phone or email:

ayende@ayende.com

+972 52-548-6969

, @ Q c

Posts: 5,953 | Comments: 44,408

filter by tags archive

The next BIG Thing in RavenDB 1.2


Is something that you probably wouldn’t even notice, to be perfectly honest. We are going to work on our map/reduce implementation.

This is freakishly complex, because we need to do updatable, persistent map/reduce. It got so complex that I decided that I can’t really implement this on my own in the RavenDB solution and moved to spiking the solution in isolation.

If you care, you can look at this here. There would be additional optimizations to worry about in RavenDB, but it is pretty much all there, in less than 400 lines of code.

I couldn’t find anything out there which was nearly as useful. Most of the map/reduce implementations are about distributing the work load and scheduling it. None of them really deal with the notion of updatable map/reduce results.

Note that the Storage layer there is both only there for the sole purpose of actually showing we can persist and restart from any point and also has critical behavior in its behavior (for example, scheduling).

I’ll probably do a set of posts about this, but for now, here is the source, have fun poking at it: https://github.com/ayende/updatable-persistent-map-reduce


Comments

Brian

Really?

static void Main() { foreach (var directory in Directory.GetDirectories(".")) { try { Directory.Delete(directory, true); } catch (Exception) { } <snip...> }

Nicolas

Take a look at this: https://github.com/nathanmarz/storm/

Maybe is not what you are looking for, or in the same language for that matter, but maybe it can give you some ideas... Or maybe not!

Daniel Lang

One shouldn't run this exe from C:/ with admin privileges... lol

Ayende Rahien

Brian, Yes... ? This is there to make sure that we clear old results from the previous run.

Starfish

I trust there are no "breaking" changes?

Ayende Rahien

Starfish, Not outward facing, not from this.

Brian

Oren, then might I suggest you at least put some sanity check around that block. Even though this is not production code, that's just leaving a loaded gun lying around (it's irresponsible coding and if you were reviewing any code that did this I'll bet you'd have blasted the author, too).

Ayende Rahien

Brian, That code is there to make my life easier. This project is there as a POC. I am not going to worry about it.

Matt Warren

"I couldn’t find anything out there which was nearly as useful. Most of the map/reduce implementations are about distributing the work load and scheduling it. None of them really deal with the notion of updatable map/reduce results."

You might want to take a look at Percolator, it's Google enhancement to Map/Reduce to solve this exact issue. I.e. so it doesn't have to re-create it's index from scratch every 2 weeks, it can do incremental updates.

See http://www.theregister.co.uk/2010/09/24/googlepercolator/ and http://static.googleusercontent.com/externalcontent/untrusted_dlcp/research.google.com/en//pubs/archive/36726.pdf

It might have some ideas that could be used in RavenDB

Ayende Rahien

Matt, I was mostly interested in going over source code, to see how the split & merge the work

Comment preview

Comments have been closed on this topic.

FUTURE POSTS

No future posts left, oh my!

RECENT SERIES

  1. The RavenDB Comic Strip (3):
    28 May 2015 - Part III – High availability & sleeping soundly
  2. Special Offer (2):
    27 May 2015 - 29% discount for all our products
  3. RavenDB Sharding (3):
    22 May 2015 - Adding a new shard to an existing cluster, splitting the shard
  4. Challenge (45):
    28 Apr 2015 - What is the meaning of this change?
  5. Interview question (2):
    30 Mar 2015 - fix the index
View all series

Syndication

Main feed Feed Stats
Comments feed   Comments Feed Stats