Ayende @ Rahien

My name is Oren Eini
Founder of Hibernating Rhinos LTD and RavenDB.
You can reach me by phone or email:


+972 52-548-6969

, @ Q c

Posts: 6,026 | Comments: 44,842

filter by tags archive

Think about production, silly!

time to read 2 min | 269 words

We just finished doing a big optimization in RavenDB, and one of the things that we needed to do was to store additional (internal) information so we could act upon it later on. If you must know, we now keep track of stats during indexing and can select the appropriate indexing approach based on the amount of data that we have available.

The details about this aren’t that important. What is important is that this is a piece of data that is used by RavenDB to make decisions. That means that just about the worst thing that we could possibly do is leave things at this state:

Think about what will happen in production, when you have an annoyed (and tired) ops team trying to figure out what is going on. Having a black box is the worst thing that you could possibly do, because you give the admin absolutely no input. And remember, you are going to be the one on call when the support phone rings.

One of the very final touches that we did was to add a debug endpoint that will expose those details to the user, so we could actually inspect them at runtime, and in production.  We have a lot of those, some are intended for monitoring purposes, such as the /admin/stats or the /databases/db-name/stats endpoints, some are meant for troubleshooting, such as the /databases/db-name/logs?type=error endpoint and some are purely for debugging purposes, such as /databases/db-name/indexes/index/name?debug=keys which gives you the stats about all the keys in a map/reduce index.

Trust me, you are going to need those, at some point.


Matthew Bonig

Is there a cheatsheet or reference somewhere of these various debugging endpoints? I can see this being incredibly useful to have a list of options I can start looking at. Didn't see anything in the docs.

Gene Hughson

"Trust me, you are going to need those, at some point."

Indeed. Sometimes things get YAGNI'ed just because the affected stakeholder isn't represented. Not designing for operations is the new not designing for exceptions.

Ayende Rahien

Matthew, They are scheduled to be documented, for now, you can see them here: http://issues.hibernatingrhinos.com/issue/RDoc-50

Sergey Shumov

Good post, one can certainly learn a couple of good practices from RavenDB. Speaking of RavenDB cluster, how would you implement logs (and stats) aggregation? Simple HTTP GET -> Concat -> Display or something exotic like UDP broadcasting?

Ayende Rahien

Sergey, Yes, you can do that by getting the stats from all servers, yes.

Comment preview

Comments have been closed on this topic.


No future posts left, oh my!


  1. Technical observations from my wife (3):
    13 Nov 2015 - Production issues
  2. Production postmortem (13):
    13 Nov 2015 - The case of the “it is slow on that machine (only)”
  3. Speaking (5):
    09 Nov 2015 - Community talk in Kiev, Ukraine–What does it take to be a good developer
  4. Find the bug (5):
    11 Sep 2015 - The concurrent memory buster
  5. Buffer allocation strategies (3):
    09 Sep 2015 - Bad usage patterns
View all series


Main feed Feed Stats
Comments feed   Comments Feed Stats