Ayende @ Rahien

Hi!
My name is Oren Eini
Founder of Hibernating Rhinos LTD and RavenDB.
You can reach me by phone or email:

ayende@ayende.com

+972 52-548-6969

, @ Q c

Posts: 18 | Comments: 82

filter by tags archive

Raven Streams architecture thoughts: Storage

time to read 5 min | 890 words

Raven Streams is a working title.

So far, I have been working mostly on getting things working from an infrastructure perspective. That is, I want to be able to successfully write & read from the streams, and I have a general architecture for it.

Basically, we have something like this:

image

This is a single stream (of which we expect to have more than one, but not a very high number). The stream makes all writes to a mem table, backed by a log. When the mem table is full, we switch to a new memtable and write all the data to a sorted string table (SST). This is very much like the design of leveldb.

Internally, however, we represent all data as etags. That is, you don’t have an actual key that you can use, the stream will generate an etag for you. And the SST are sorted in order, so the oldest events are in the oldest file, etc.

On startup, if we haven’t flushed properly, we will translate the log file into a new SST and start from scratch. I am not sure yet if we want / need to implement compaction. Maybe we will do that for very small files only.

Right now there are two methods that you actually care about:

  • AppendAsync(data)
  • ReadFrom(etag)

This is how you use them:

   1: for (int i = 0; i < 15; i++)
   2: {
   3:     eventStream.AppendAsync(new RavenJObject { { "counter", i } }).Wait();
   4: }
   5:  
   6: int x = 0;
   7: foreach (var item in eventStream.ReadFrom(Etag.Empty))
   8: {
   9:     var value = item.Value<int>("counter");
  10:     Assert.Equal(x++, value);
  11: }

This is all working pretty nicely right now. It isn’t nearly ready for production or even test usage, but this is encouraging. Next, I’ll discuss the intended implementation of map/reduce against this.


Comments

Daniel Marbach

Oren, why don't you just hire greg youngs event store team and use their knowledge to build blazing fast storage engine. Performance vise I doubt your solution can compete with that. Why reinvent the wheel?

Ayende Rahien

Daniel, Because we aren't talking about the same thing. I am not trying to create an event store.

Daniel Marbach

Yeah but you need fast and transactional appen only store or not? That is present there

Ayende Rahien

Daniel, Different things altogether. One append only store doesn't match the needs of another.

Comment preview

Comments have been closed on this topic.

FUTURE POSTS

  1. The insidious cost of allocations - 8 hours from now
  2. Buffer allocation strategies: A possible solution - 3 days from now
  3. Buffer allocation strategies: Explaining the solution - 4 days from now
  4. Buffer allocation strategies: Bad usage patterns - 5 days from now
  5. The useless text book algorithms - 6 days from now

And 1 more posts are pending...

There are posts all the way to Sep 11, 2015

RECENT SERIES

  1. Find the bug (5):
    20 Apr 2011 - Why do I get a Null Reference Exception?
  2. Production postmortem (10):
    03 Sep 2015 - The industry at large
  3. What is new in RavenDB 3.5 (7):
    12 Aug 2015 - Monitoring support
  4. Career planning (6):
    24 Jul 2015 - The immortal choices aren't
View all series

Syndication

Main feed Feed Stats
Comments feed   Comments Feed Stats