Ayende @ Rahien

My name is Oren Eini
Founder of Hibernating Rhinos LTD and RavenDB.
You can reach me by phone or email:


+972 52-548-6969

, @ Q c

Posts: 6,128 | Comments: 45,548

filter by tags archive

API DesignSharding Status for failure scenarios–explicit failure management

time to read 2 min | 276 words

Still going on with the discussion on how to handle failures in a sharded cluster, we are back to the question of how to handle the scenario of one node in a cluster going down. The question is, what should be the system behavior in such a scenario.

In the previous post, we discussed the option of simply ignoring the failure, and the option of simply failing entirely. Both options are unpalatable, because we either transparently hide some data from the user (which reside on the failing node) or we take the entire system down when a single node is down.

Another option that was suggested in the mailing list is to actually expose this to the user, like so:

ShardingStatus status;
va recentPosts = session.Query<Post>()
          .ShardingStatus( out status )

This will give us the status information about potentially failing shards.

I intensely dislike this option, and I’ll discuss the reasons why on the next post. In the meantime, I would like to hear your opinion about this API choice.

More posts in "API Design" series:

  1. (20 Jul 2015) We’ll let the users sort it out
  2. (17 Jul 2015) Small modifications over a network
  3. (01 Jun 2012) Sharding Status for failure scenarios–Solving at the right granularity
  4. (31 May 2012) Sharding Status for failure scenarios–explicit failure management doesn’t work
  5. (30 May 2012) Sharding Status for failure scenarios–explicit failure management
  6. (29 May 2012) Sharding Status for failure scenarios–ignore and move on
  7. (28 May 2012) Sharding Status for failure scenarios



This approach could be ok, if omitting the method would cause an exception (a method name like AllowPartialResults() could be slightly better). It would be easier to implement than catching specific exception (where the partial results are in the exception details) - the solution that was proposed in the last post comments. But it would still ensure that the developer has to make a concious decision that the data he is retrieving is allowed to be incomplete (which is ok for blog posts, but is not ok when calculating financials).

This approach could also enable certain entities to specify this automatically when the Query is called so that the decision making is left to the author of the model instead of the consumer.


One of the ways could be storing something like a List inside a session (or maybe even populating it to the Store), so every careful site-owner could display a big red cross on the top-right corner of the site, meaning that something bad happened :)

That requires no friction at every .Query() call, but will give a sensible information about an error with only one-time setup.


I meant List<Error> (List of Error) but the parser broken my c# :)


The desired API depends on the application. Sometimes you'll want to silently ignore shard's failure and sometimes you'll explicitly handle it. But it's not an error, it's a normal situation that some shard may be unavailable, therefore Ravens API shouldn't throw an exception. This approach (with ShardingStatus out parameter) is better than an exception but it's not very elegant as it requires you to remember to call ShardingStatus method with each query and then to add some code for handling the status returned. Besides, the out parameter is not so great for fluent interface because you don't know when it will be set. A callback function would be imho better.


update: wrong, the out parameter can't be used here because it needs to be returned after the query is executed, not before


It's wrong if you want to make users use it always on per query basis. Raven should allow introducing a cross-cutting setting, registered once, to handle this situation (like in ISessionFactory, if there is one) and overriding when it's needed. Handling majority of cases in one way is what you should go for.


What about a Maybe-like monad? I mean a session.ShardQuery method could return a enhanced type, so that the user must explicitely get the underlying collection by matching if it is partial or not. The strategy to apply next is up to him.


Is there a possibility of taking some design ideas from a RAID and build the sharding out in a way that if a server goes down the remaining machines in the cluster can rebuild themselves to return full results if space allows?


How is this any different than a stale index? Raven DB already has a way to communicate that your query results may be incomplete. The reason the results are incomplete is really secondary, either way you have incomplete results that can cause business logic issues.

Just use IsStale and WaitForNonStaleResults and add something to RavenQueryStatistics to describe the stale reason(still indexing or shard down or ...)

Think of it this way how should the application handle a down shard vs a long running index process? They both cause missing results for an indeterminate amount of time and the application should respond the same regardless by either waiting to see if the results become complete or failing the operation and notifying the user.

Martin Doms

An extension method with a side effect actually makes me slightly sick to my stomach.

Patrick Huizinga

@Martin Doms, Where did you get the idea the discussion was about an extension method?

Patrick Huizinga

@Justin, The big difference between a stale index and a down shard, is that the index is expected to catch up quickly (< 1 sec.), while a down shard is 'expected' to remain unavailable for a while (> 1 min.).

So there is no danger waiting for the index to catch up, while it's a bad idea to wait (block) for the shard to come back.


@Patrick, If there is no danger in waiting for a index to catch up why is the default not to wait and return incomplete results?

Indexes being rebuilt on large databases can take quite a while(>1 min) so the "danger" of waiting on a re-index maybe be just as bad as waiting for a down shard to come up.

Either way Raven already provides a boolean status of possible incomplete results on a query and RavenQueryStatistics that can be extended to describe in more detail why those results are incomplete.

Ayende Rahien

Justin, Because that would mean waiting. It means that you have to stop and wait for a result and that may increase your latency. Also, that depend on what type of waiting you are doing. But in general, showing results from a few ms ago is more than good enough.


That's why Raven doesn't wait by default right? What does it matter to the user/application why the results are incomplete? It probably matters a lot to the admin but either way the user/application didn't get the expected results and can't make certain application level decisions until it does, and may not for an unknown amount of time.

If the DB has recently been loaded from an ETL process, the indexing may take hours which is probably why WaitForNonStaleResults has a timeout right? All this has already been handled.

I would imagine whatever you do for a down shard will look very similar to how a stale index is handled currently since you want the same tenets to apply (system/world doesn't stop, no waiting).

Ayende Rahien

Justin, Staleness that takes hours to go away is REALLY rare. We usually talking about ms under normal load, seconds under very heavy load. And there is a big difference between "those results are accurate as of TIME" vs. "those results may be partial".


I would hope shards going down are just as rare ;).

Both issues are time based, specifically transaction time, in both situations the transaction has already occurred in the past and the index is not showing the committed transaction for two different technical reasons, but logically the issue is identical to the application.

Regardless of how long it takes for the indexing to complete or the shard to come up, the the application must handle these situations in a similar manner.

If you code your application to assume indexing only takes <1 second and then a large amount of data is re-indexed what happens? You must handle this possibility somehow. Once you've handled the long-running index operation, you've just handled a down shard too, at least for queries against indexes.

Hendry Luk

What about the standard .net TryXxx(out) API convention, which will do almost the exact behavior as its Xxx() counterpart, except that it will return its success/failure result in lieau of exceptions?

Comment preview

Comments have been closed on this topic.


  1. The low level interview question - 3 hours from now
  2. The worker pattern - 3 days from now

There are posts all the way to May 30, 2016


  1. The design of RavenDB 4.0 (14):
    26 May 2016 - The client side
  2. RavenDB 3.5 whirl wind tour (14):
    25 May 2016 - Got anything to declare, ya smuggler?
  3. Tasks for the new comer (2):
    15 Apr 2016 - Quartz.NET with RavenDB
  4. Code through the looking glass (5):
    18 Mar 2016 - And a linear search to rule them
  5. Find the bug (8):
    29 Feb 2016 - When you can't rely on your own identity
View all series



Main feed Feed Stats
Comments feed   Comments Feed Stats