Ayende @ Rahien

My name is Oren Eini
Founder of Hibernating Rhinos LTD and RavenDB.
You can reach me by phone or email:


+972 52-548-6969

, @ Q c

Posts: 6,125 | Comments: 45,492

filter by tags archive

Dogfood sometimes doesn’t taste so good

time to read 2 min | 329 words

We just finished rolling back our internal servers migration to 3.0 back to 2.5. That was quite unpleasant, and was actually noticed by users.

That isn’t pleasant, but it is always better if we get the egg all our face than if it is a customer. The actual issue that we run into was pretty interesting.

The problem is that the database we use for running this blog (as well as most of our internal systems) has been through… a lot. It has gone through pretty much every released version, and many that weren’t actually released.

That means that from storage perspective (only of interest to RavenDB developers), it is a bit of a mess. That in turn meant that we had to do extra work to convert the storage from the 2.5 version to the 3.0 version. That caused enough memory to be used that we hit our limits on memory usage, and failed to convert it to the 3.0 version.

That meant that it was stuck. That is actually one of the reasons that we test those things on our own systems, so that was great.

The not so great part was that we also uncovered another interesting bug (actually, several of them in conjunction). The new studio had a tendency to read the stats from all the available databases, if the number we had was small enough. That was done so we can show the number of documents on each database in the databases page.

That meant that we would effectively start all of them in parallel (and consume resources that weren’t actually needed).

And that, in turn, exposed a race condition (in Esent!) that can resulted in a hard process crash. That was the hardest thing to get over with, because obviously we don’t have source access to Esent and it was kinda of hard to pinpoint where this was actually happening and why.

All fixed and good now, and ready to try again.



Interesting. Do you have any plans of migrating to Voron on production?

Ayende Rahien

Rodrigo, Yes, that is in the plan. As you can imagine, we are working through issues now, and I want to make sure that the Esent version is stable before trying Voron out.




sometimes, you can impersonate other users just by typing their email address.


are you hiring QAs as well?

Comment preview

Comments have been closed on this topic.


  1. RavenDB 3.5 whirl wind tour: I'll have the 3+1 goodies to go, please - 3 days from now
  2. The design of RavenDB 4.0: Voron has a one track mind - 4 days from now
  3. RavenDB 3.5 whirl wind tour: Digging deep into the internals - 5 days from now
  4. The design of RavenDB 4.0: Separation of indexes and documents - 6 days from now
  5. RavenDB 3.5 whirl wind tour: Deeper insights to indexing - 7 days from now

And 10 more posts are pending...

There are posts all the way to May 30, 2016


  1. The design of RavenDB 4.0 (14):
    05 May 2016 - Physically segregating collections
  2. RavenDB 3.5 whirl wind tour (14):
    04 May 2016 - I’ll find who is taking my I/O bandwidth and they SHALL pay
  3. Tasks for the new comer (2):
    15 Apr 2016 - Quartz.NET with RavenDB
  4. Code through the looking glass (5):
    18 Mar 2016 - And a linear search to rule them
  5. Find the bug (8):
    29 Feb 2016 - When you can't rely on your own identity
View all series


Main feed Feed Stats
Comments feed   Comments Feed Stats