Ayende @ Rahien

Hi!
My name is Oren Eini
Founder of Hibernating Rhinos LTD and RavenDB.
You can reach me by phone or email:

ayende@ayende.com

+972 52-548-6969

, @ Q c

Posts: 10 | Comments: 35

filter by tags archive

Dogfood sometimes doesn’t taste so good

time to read 2 min | 329 words

We just finished rolling back our internal servers migration to 3.0 back to 2.5. That was quite unpleasant, and was actually noticed by users.

That isn’t pleasant, but it is always better if we get the egg all our face than if it is a customer. The actual issue that we run into was pretty interesting.

The problem is that the database we use for running this blog (as well as most of our internal systems) has been through… a lot. It has gone through pretty much every released version, and many that weren’t actually released.

That means that from storage perspective (only of interest to RavenDB developers), it is a bit of a mess. That in turn meant that we had to do extra work to convert the storage from the 2.5 version to the 3.0 version. That caused enough memory to be used that we hit our limits on memory usage, and failed to convert it to the 3.0 version.

That meant that it was stuck. That is actually one of the reasons that we test those things on our own systems, so that was great.

The not so great part was that we also uncovered another interesting bug (actually, several of them in conjunction). The new studio had a tendency to read the stats from all the available databases, if the number we had was small enough. That was done so we can show the number of documents on each database in the databases page.

That meant that we would effectively start all of them in parallel (and consume resources that weren’t actually needed).

And that, in turn, exposed a race condition (in Esent!) that can resulted in a hard process crash. That was the hardest thing to get over with, because obviously we don’t have source access to Esent and it was kinda of hard to pinpoint where this was actually happening and why.

All fixed and good now, and ready to try again.


Comments

Rodrigo

Interesting. Do you have any plans of migrating to Voron on production?

Ayende Rahien

Rodrigo, Yes, that is in the plan. As you can imagine, we are working through issues now, and I want to make sure that the Esent version is stable before trying Voron out.

Chris

sometimes?

chris

sometimes, you can impersonate other users just by typing their email address.

chris

are you hiring QAs as well?

Comment preview

Comments have been closed on this topic.

FUTURE POSTS

  1. Production postmortem: The case of the memory eater and high load - about one day from now
  2. Production postmortem: The case of the lying configuration file - 3 days from now
  3. Production postmortem: The industry at large - 4 days from now
  4. The insidious cost of allocations - 5 days from now
  5. Find the bug: The concurrent memory buster - 6 days from now

And 4 more posts are pending...

There are posts all the way to Sep 10, 2015

RECENT SERIES

  1. Find the bug (5):
    20 Apr 2011 - Why do I get a Null Reference Exception?
  2. Production postmortem (10):
    14 Aug 2015 - The case of the man in the middle
  3. What is new in RavenDB 3.5 (7):
    12 Aug 2015 - Monitoring support
  4. Career planning (6):
    24 Jul 2015 - The immortal choices aren't
View all series

Syndication

Main feed Feed Stats
Comments feed   Comments Feed Stats