Ayende @ Rahien

Refunds available at head office

The subtle distinction between snapshot isolation and read committed

I am using db transaction isolation levels for a reason here, they make it easier to reason about what is going on.

In short, RavenDB currently supports two storage engine options, Esent and Munin. Esent is what we usually use for production, and Munin is usually used for testing. We wrote Munin as a transactional, fully managed, storage engine a while ago. And it has mostly served us well, but Esent is what we usually aim for. That is the production use case.

We recently made a few changes that resulted in test failures on Munin, only in one run out of two dozens or so, but always worked with Esent.

Naturally, because of the random nature of the problem, I suspected the issue being a race condition in Munin. That happened in the past ,and obviously they are very hard to root out completely. But after finally isolating everything down to a simple test case (writing to two “tables” with associated information), I finally figured it out.

Munin is working just fine, it hasn’t got a spec of a problem. It is just that, when we built it, I built it to support Read Commited Isolation Level. While Esent is providing Snapshot isolation level. The code assumes snapshot isolation level at some pretty level. Obviously, this sort of thing shows up as a race condition, and it is extremely hard to debug, as anyone who ever dealt with those issues in RDBMS can testify.

So my task now is not to fix a bug in Munin, but to actually implement snapshot isolation. As it turned out, actually moving Munin from read committed isolation to snapshot isolation was a lot easier than finding the problem.

I am torn between being pleased that I found the issue, happy that Munin doesn’t have a bug and pissed that it took me that long.

Tags:

Posted By: Ayende Rahien

Published at

Originally posted at

Comments

Sean Kearon
09/18/2012 10:37 AM by
Sean Kearon

So, does that move Munin closer to being suitable for production?

John Bloom
09/18/2012 01:29 PM by
John Bloom

"I am torn between being pleased that I found the issue, happy that Munin doesn’t have a bug and pissed that it took me that long."

I think that developers have to deal with this emotion on a regular basis. And "taking a long time" is relative to the task at hand. Some problems taking 5 minutes is too long and I kick myself for not figuring it out quicker.

edward
09/18/2012 07:25 PM by
edward

why have and maintain 2 storage engines while only one is aimed for production?

Duckie
09/18/2012 08:39 PM by
Duckie

Munin is written in .net, while esent is a part of windows. The goal, from what i know, is to have a independent storage engine. Munin is just not ready for production. Yet.

http://en.wikipedia.org/wiki/ExtensibleStorageEngine http://ayende.com/blog/4686/raven-munin

Daniel Lang
09/18/2012 10:45 PM by
Daniel Lang

@Edward: Because Munin can run in-memory, which is actually very handy from time to time (i.e. unit-testing).

Matt Warren
09/18/2012 10:48 PM by
Matt Warren

@Edward

One reason is to have a way to run on Linux, although a version running on BDB might be a better option for that (see https://groups.google.com/d/topic/ravendb/WhuwS218-xg/discussion).

But the main reason for Munin at the moment is so that RavenDB can run in-memory, for unit tests.

Pure Krome
09/18/2012 11:20 PM by
Pure Krome

:golf clap: :)

Edward
09/19/2012 02:08 PM by
Edward

@Daniel, @Matt thanks! [P.S. Nice blogs!]

Comments have been closed on this topic.