Reviewing LevelDBPart III, WriteBatch isn’t what you think it is
One of the key external components of leveldb is the idea of WriteBatch. It allows you to batch multiple operations into a single atomic write.
It looks like this, from an API point of view:
As we have learned in the previous post, WriteBatch is how leveldb handles all writes. Internally, any call to Put or Delete is translated into a single WriteBatch, then there is some batching involved across multiple batches, but that is beside the point right now.
I dove into the code for WriteBatch, and immediately I realized that this isn’t really what I bargained for. In my mind, WriteBatch was supposed to be something like this:
Which would hold the in memory operations until they get written down to disk, or something.
Instead, it appears that leveldb took quite a different route. The entire data is stored in the following format:
This is the in memory value, mind. So we are already storing this in a single buffer. I am not really sure why this is the case, to be honest.
WriteBatch is pretty much a write only data structure, with one major exception:
You can iterate over the batch. The problem is that we now have this implementation for Iterate:
So we write it directly to a buffer, then read from that buffer. The interesting bit is that the actual writing to leveldb itself is done in a similar way, see:
As I can figure it so far, we have the following steps:
- WriteBatch.Put / WriteBatch.Delete gets called, and the values we were sent are copied into our buffer.
- We actually save the WriteBatch, at which point we unpack the values out of the buffer and into the memtable.
It took me a while to figure it out, but I think that I finally got it. The reason this is the case is that leveldb is a C++ application. As such, memory management is something that it needs to worry about explicitly.
In particular, you can’t just rely on the memory you were passed to be held, the user may release that memory after they called to Put. This means, in turn, that you must copy the memory to memory that leveldb allocated, so leveldn can manage its own lifetime. This is a foreign concept to me because it is such a strange thing to do in the .NET land, where memory cannot just disappear underneath you.
On my next post, I’ll deal a bit more with this aspect, buffers management and memory handling in general.
More posts in "Reviewing LevelDB" series:
- (26 Apr 2013) Part XVIII–Summary
- (15 Apr 2013) Part XVII– Filters? What filters? Oh, those filters…
- (12 Apr 2013) Part XV–MemTables gets compacted too
- (11 Apr 2013) Part XVI–Recovery ain’t so tough?
- (10 Apr 2013) Part XIV– there is the mem table and then there is the immutable memtable
- (09 Apr 2013) Part XIII–Smile, and here is your snapshot
- (05 Apr 2013) Part XI–Reading from Sort String Tables via the TableCache
- (04 Apr 2013) Part X–table building is all fun and games until…
- (02 Apr 2013) Part VIII–What are the levels all about?
- (29 Mar 2013) Part VII–The version is where the levels are
- (28 Mar 2013) Part VI, the Log is base for Atomicity
- (27 Mar 2013) Part V, into the MemTables we go
- (26 Mar 2013) Part IV
- (22 Mar 2013) Part III, WriteBatch isn’t what you think it is
- (21 Mar 2013) Part II, Put some data on the disk, dude