Ayende @ Rahien

Refunds available at head office

Implications of design decisions: Read Striping

When using RavenDB replication, you have the option to do something that is called “Read Striping”. Instead of using replication only for High Availability and Disaster Recovery, we can also spread our reads among all the replicating servers.

The question then becomes, how would you select which server to use for which read request?

The obvious answer is to do something like this:

Request Number % Count of Servers = current server index.

This is simple to understand, easy to implement and oh so horribly wrong in a subtle way that it is scary.

Let us look at the following scenario, shall we?

image

  • Session #1 loads users/3 from Server A
  • Session #1 then query (with non stale results) from Server B
  • The users/3 document hasn’t been replicated to Server B yet, so Server B sends back a reply “here are my (non stale) results)”
  • Session #1 assumes that users/3 must be in the results, since they are non stale, and blows up as a result.

We want to avoid such scenarios.

Werner Vogels, Amazon’s CTO, has the following consistency definitions:

  • Causal consistency. If process A has communicated to process B that it has updated a data item, a subsequent access by process B will return the updated value and a write is guaranteed to supersede the earlier write. Access by process C that has no causal relationship to process A is subject to the normal eventual consistency rules.
  • Read-your-writes consistency. This is an important model where process A after it has updated a data item always accesses the updated value and never will see an older value. This is a special case of the causal consistency model.
  • Session consistency. This is a practical version of the previous model, where a process accesses the storage system in the context of a session. As long as the session exists, the system guarantees read-your-writes consistency. If the session terminates because of certain failure scenarios a new session needs to be created, and the guarantees do not overlap the sessions.
  • Monotonic read consistency. If a process has seen a particular value for the object any subsequent accesses will never return any previous values.
  • Monotonic write consistency. In this case the system guarantees to serialize the writes by the same process. Systems that do not guarantee this level of consistency are notoriously hard to program.

What is the consistency model offered by this?

Request Number % Count of Servers = current server index.

As it turned out, it is pretty much resolves into “none”. Because each request may be re-directed into a different server, there is no consistency guarantees that we can make.  That can make reasoning about the system really hard.

Instead, RavenDB choose to go another route, and we use the following formula to calculate which server we will use.

Session Number % Count of Servers = current server index.

By using a session, rather than a request, counter to decide which server we will use when spreading reads, we ensure that all of the requests made within a single session are always going to the same server, ensuring we have session consistency. Within the scope of a single session, we can rely on that consistency, instead of the chaos of no consistency at all.

It is funny, how such a small change can have such profound implication.

Tags:

Posted By: Ayende Rahien

Published at

Originally posted at

Comments

Simon Hughes
07/17/2012 09:43 AM by
Simon Hughes

Simple solutions are always the best. Where does the session number come from? Is that similar a connection ID? What about using a website to update/read data. i.e. Could a load balancer affect your session number, by spreading your access over two webservers?

Ayende Rahien
07/17/2012 09:46 AM by
Ayende Rahien

Simon, Session id comes from a simple int32 that gets incremented on every session. It is not a connection id.

Writes usually goes to the master, always, and the way RavenDB works, you do all the reads up front, then do a single write.

Rafal
07/17/2012 11:31 AM by
Rafal

but if you have a session per http request then the next request from the same user is quite likely to go to another server, isn't it? And maybe there will be no 'kaboom' but the user will be quite surprised when he won't see the changes he has just made...

Karg
07/17/2012 03:37 PM by
Karg

The Session is an object that you create in code. It lives until you dispose of it. Therefore, you have control over the duration of your "consistency".

João P. Bragança
07/17/2012 04:43 PM by
João P. Bragança

Rafal,

That would only be an issue if the user spent less time on the screen than it took for replication to occur. And even so, usually it isn't a big deal. We don't always see our comments on the blog post right after we make them.

peter
07/17/2012 04:47 PM by
peter

"Systems that do not guarantee this level of consistency are notoriously hard to program."

I am sure thart should say "Systems that guarantee this level of consistency are notoriously hard to program." ?

peter
07/17/2012 05:03 PM by
peter

OK, I read up a little on Monotonic writes and what the sentence means is that programming against systems without this guarantee is notoriously difficult.

Rafal
07/17/2012 07:37 PM by
Rafal

Joao, not everyone is doing blogs ;) This 'session consistency' is a very weak guarantee for web applications when subsequent requests are very likely to be served from different replicas. Also, this approach requires bidirectional replication with very low latency, which will likely cause problems. 'User session' consistency would be a much better guarantee (the same database handling all requests from an user/web client). For example, some load balancers use client IP address to direct that client's request to same web server.

Ayende Rahien
07/18/2012 06:14 AM by
Ayende Rahien

Rafal, You have control over that as a user, you can decide whatever to pass the same replication base whenever using the same user.

Comments have been closed on this topic.