+972 52-548-6969
API DesignSharding Status for failure scenarios–explicit failure management
Still going on with the discussion on how to handle failures in a sharded cluster, we are back to the question of how to handle the scenario of one node in a cluster going down. The question is, what should be the system behavior in such a scenario.
In the previous post, we discussed the option of simply ignoring the failure, and the option of simply failing entirely. Both options are unpalatable, because we either transparently hide some data from the user (which reside on the failing node) or we take the entire system down when a single node is down.
Another option that was suggested in the mailing list is to actually expose this to the user, like so:
ShardingStatus status;
va recentPosts = session.Query<Post>()
.ShardingStatus( out status )
.OrderByDescending(x=>x.PublishedAt)
.Take(20)
.ToList();
This will give us the status information about potentially failing shards.
I intensely dislike this option, and I’ll discuss the reasons why on the next post. In the meantime, I would like to hear your opinion about this API choice.
More posts in "API Design" series:
- (04 Dec 2017) The lack of a method was intentional forethought
- (27 Jul 2016) robust error handling and recovery
- (20 Jul 2015) We’ll let the users sort it out
- (17 Jul 2015) Small modifications over a network
- (01 Jun 2012) Sharding Status for failure scenarios–Solving at the right granularity
- (31 May 2012) Sharding Status for failure scenarios–explicit failure management doesn’t work
- (30 May 2012) Sharding Status for failure scenarios–explicit failure management
- (29 May 2012) Sharding Status for failure scenarios–ignore and move on
- (28 May 2012) Sharding Status for failure scenarios
Comments
This approach could be ok, if omitting the method would cause an exception (a method name like AllowPartialResults() could be slightly better). It would be easier to implement than catching specific exception (where the partial results are in the exception details) - the solution that was proposed in the last post comments. But it would still ensure that the developer has to make a concious decision that the data he is retrieving is allowed to be incomplete (which is ok for blog posts, but is not ok when calculating financials).
This approach could also enable certain entities to specify this automatically when the Query<T> is called so that the decision making is left to the author of the model instead of the consumer.
One of the ways could be storing something like a List<Error> inside a session (or maybe even populating it to the Store), so every careful site-owner could display a big red cross on the top-right corner of the site, meaning that something bad happened :)
That requires no friction at every .Query() call, but will give a sensible information about an error with only one-time setup.
I meant List<Error> (List of Error) but the parser broken my c# :)
The desired API depends on the application. Sometimes you'll want to silently ignore shard's failure and sometimes you'll explicitly handle it. But it's not an error, it's a normal situation that some shard may be unavailable, therefore Ravens API shouldn't throw an exception. This approach (with ShardingStatus out parameter) is better than an exception but it's not very elegant as it requires you to remember to call ShardingStatus method with each query and then to add some code for handling the status returned. Besides, the out parameter is not so great for fluent interface because you don't know when it will be set. A callback function would be imho better.
update: wrong, the out parameter can't be used here because it needs to be returned after the query is executed, not before
It's wrong if you want to make users use it always on per query basis. Raven should allow introducing a cross-cutting setting, registered once, to handle this situation (like in ISessionFactory, if there is one) and overriding when it's needed. Handling majority of cases in one way is what you should go for.
What about a Maybe-like monad? I mean a session.ShardQuery method could return a enhanced type, so that the user must explicitely get the underlying collection by matching if it is partial or not. The strategy to apply next is up to him.
Is there a possibility of taking some design ideas from a RAID and build the sharding out in a way that if a server goes down the remaining machines in the cluster can rebuild themselves to return full results if space allows?
How is this any different than a stale index? Raven DB already has a way to communicate that your query results may be incomplete. The reason the results are incomplete is really secondary, either way you have incomplete results that can cause business logic issues.
Just use IsStale and WaitForNonStaleResults and add something to RavenQueryStatistics to describe the stale reason(still indexing or shard down or ...)
Think of it this way how should the application handle a down shard vs a long running index process? They both cause missing results for an indeterminate amount of time and the application should respond the same regardless by either waiting to see if the results become complete or failing the operation and notifying the user.
An extension method with a side effect actually makes me slightly sick to my stomach.
@Martin Doms, Where did you get the idea the discussion was about an extension method?
@Justin, The big difference between a stale index and a down shard, is that the index is expected to catch up quickly (< 1 sec.), while a down shard is 'expected' to remain unavailable for a while (> 1 min.).
So there is no danger waiting for the index to catch up, while it's a bad idea to wait (block) for the shard to come back.
@Patrick, If there is no danger in waiting for a index to catch up why is the default not to wait and return incomplete results?
Indexes being rebuilt on large databases can take quite a while(>1 min) so the "danger" of waiting on a re-index maybe be just as bad as waiting for a down shard to come up.
Either way Raven already provides a boolean status of possible incomplete results on a query and RavenQueryStatistics that can be extended to describe in more detail why those results are incomplete.
Justin, Because that would mean _waiting_. It means that you have to stop and wait for a result and that may increase your latency. Also, that depend on what type of waiting you are doing. But in general, showing results from a few ms ago is more than good enough.
That's why Raven doesn't wait by default right? What does it matter to the user/application why the results are incomplete? It probably matters a lot to the admin but either way the user/application didn't get the expected results and can't make certain application level decisions until it does, and may not for an unknown amount of time.
If the DB has recently been loaded from an ETL process, the indexing may take hours which is probably why WaitForNonStaleResults has a timeout right? All this has already been handled.
I would imagine whatever you do for a down shard will look very similar to how a stale index is handled currently since you want the same tenets to apply (system/world doesn't stop, no waiting).
Justin, Staleness that takes hours to go away is REALLY rare. We usually talking about ms under normal load, seconds under very heavy load. And there is a big difference between "those results are accurate as of TIME" vs. "those results may be partial".
I would hope shards going down are just as rare ;).
Both issues are time based, specifically transaction time, in both situations the transaction has already occurred in the past and the index is not showing the committed transaction for two different technical reasons, but logically the issue is identical to the application.
Regardless of how long it takes for the indexing to complete or the shard to come up, the the application must handle these situations in a similar manner.
If you code your application to assume indexing only takes <1 second and then a large amount of data is re-indexed what happens? You must handle this possibility somehow. Once you've handled the long-running index operation, you've just handled a down shard too, at least for queries against indexes.
What about the standard .net TryXxx(out) API convention, which will do almost the exact behavior as its Xxx() counterpart, except that it will return its success/failure result in lieau of exceptions?
Comment preview