Ayende @ Rahien

My name is Oren Eini
Founder of Hibernating Rhinos LTD and RavenDB.
You can reach me by phone or email:


+972 52-548-6969

, @ Q c

Posts: 6,128 | Comments: 45,549

filter by tags archive

API DesignSharding Status for failure scenarios–ignore and move on

time to read 2 min | 246 words

In my previous post, I discussed how to handle partial failure in sharded cluster scenario. This is particularly hard because this is the case where one node out of a 3 nodes (or more) cluster is failing, and it is entirely likely that we can give the user at least partial service properly.

The most obvious, and probably easiest option, is to simply catch and log the error for the failing server and not notify the calling application code about this. The nice thing about this option is that if you have a failing server, you don’t have your entire system goes down, and can handle this at least partially.

The sad part about this option is that there really is a good chance that you won’t notice that some part of the system is down, and that you are actually returning only partial results. That can lead to some really nasty scenarios, such as the case where we “lose” an order, or a payment, and we don’t show this to the user.

That can lead to some really frustrating scenarios where a customer is saying “but I see the payment right here” and the help desk says “I am sorry, but I don’t see it, therefor you don’t get any service, have a good day, bye”.

Still… that is probably the best case scenario considering the alternative being the entire system being unavailable if any single node is down.

Or is it… ?

More posts in "API Design" series:

  1. (20 Jul 2015) We’ll let the users sort it out
  2. (17 Jul 2015) Small modifications over a network
  3. (01 Jun 2012) Sharding Status for failure scenarios–Solving at the right granularity
  4. (31 May 2012) Sharding Status for failure scenarios–explicit failure management doesn’t work
  5. (30 May 2012) Sharding Status for failure scenarios–explicit failure management
  6. (29 May 2012) Sharding Status for failure scenarios–ignore and move on
  7. (28 May 2012) Sharding Status for failure scenarios


Thomas Krause

How about throwing an exception but including the partial results in the exception details?

Like this you force the calling code to handle this scenario. It can then show a message to the user ("due to technical problems some information may be missing") along with the partial results.

Ayende Rahien

Thomas, Wow! That is something that I would have never though of. Interesting approach, and something that I might well utilize in the future. It nicely handle the scenario of getting the data and not avoiding the error, but it doesn't actually work in practice. It means that you have to be very careful about what you are doing, and that a single place where you forgot to do this would result in your site being effectivley down.


What if put "AllNodesAreOnline" flat into statistics? In this we could find out whether query was executed correctly similar to how we get total results count for paging (http://old.ravendb.net/faq/total-results-in-paged-data-set)

Thomas Krause

Yes, that is the downside. The alternative would be to return a result always but include some error details in the result. But this makes it very easy to silently ignore the error.

It probably depends on the application which scenario is more desirable.


Hm, what about classic saving all relevant data in single shard? In such case client and all his orders/payments/etc would go to single shard and either he would get everything, either nothing, but other clients would still be able to access system.


Why not make it configurable? Safe by default would probably mean throw the exception (with the results), but give the option to change that behaviour.

Ayende Rahien

Jonty, Either options doesn't work. It is too big a tool, turn it on/off globally isn't helpful


Just an alternative view would be using additional passive mirroring as, for example Greenplum offers. I.e. you have A B C nodes, B mirrors A, C mirrors B and A mirrors C. So if one of the nodes goes offline, system can continue to work (with integrity).

At the API level this should come as additional metadata of the session and it should be controlled by initializing the session (fails with exception, does not fail with the exception). 99% of the time I can imagine, app should not know about the node down, rather than sysadmin should get the alarm shower.

Chris Wright

As a user of ravendb, I want some way I can check the global health of my database without mucking about with exception handling. I want to be able to look in my session at the end of a request and see: was there anything that might have impacted correctness?

If so, I'll throw up an alert message at the top of the page: "We might not be showing you all available data."

That way, my customer support people can see that and give people the benefit of the doubt. They have the most information I can still provide, and also know that they don't have all the information they should.

I don't necessarily need to know this for each query; that might be useful, but it's way too granular for most of what I need to do. I just need to know whether to put up the alert box or not.

In a different context, I might want more granular options. Maybe I want to fail right away, or just check at one or two critical points if I have missing data.

I think the two ways of handling I want are: - Fail immediately always, by throwing an exception. - Give me the best results possible always, and record on Session whether there are any failures. That should be a property available for me to check at any time, though if I'm using some sort of batching, I may have started an operation that is doomed and will not be reflected there.

I'm not sure of anything else that would be useful.

Pop Catalin

Either way can be the "only" good way for a certain type of applications/

Therefore how about, add the possibility to set the default globally and allow overriding locally.

But please go with the safe default ;). Those that want unsafe should opt in, not out out.

Comment preview

Comments have been closed on this topic.


  1. The worker pattern - 2 days from now

There are posts all the way to May 30, 2016


  1. The design of RavenDB 4.0 (14):
    26 May 2016 - The client side
  2. RavenDB 3.5 whirl wind tour (14):
    25 May 2016 - Got anything to declare, ya smuggler?
  3. Tasks for the new comer (2):
    15 Apr 2016 - Quartz.NET with RavenDB
  4. Code through the looking glass (5):
    18 Mar 2016 - And a linear search to rule them
  5. Find the bug (8):
    29 Feb 2016 - When you can't rely on your own identity
View all series


Main feed Feed Stats
Comments feed   Comments Feed Stats