Implications of design decisions: Read Striping

time to read 3 min | 595 words

When using RavenDB replication, you have the option to do something that is called “Read Striping”. Instead of using replication only for High Availability and Disaster Recovery, we can also spread our reads among all the replicating servers.

The question then becomes, how would you select which server to use for which read request?

The obvious answer is to do something like this:

Request Number % Count of Servers = current server index.

This is simple to understand, easy to implement and oh so horribly wrong in a subtle way that it is scary.

Let us look at the following scenario, shall we?

image

  • Session #1 loads users/3 from Server A
  • Session #1 then query (with non stale results) from Server B
  • The users/3 document hasn’t been replicated to Server B yet, so Server B sends back a reply “here are my (non stale) results)”
  • Session #1 assumes that users/3 must be in the results, since they are non stale, and blows up as a result.

We want to avoid such scenarios.

Werner Vogels, Amazon’s CTO, has the following consistency definitions:

  • Causal consistency. If process A has communicated to process B that it has updated a data item, a subsequent access by process B will return the updated value and a write is guaranteed to supersede the earlier write. Access by process C that has no causal relationship to process A is subject to the normal eventual consistency rules.
  • Read-your-writes consistency. This is an important model where process A after it has updated a data item always accesses the updated value and never will see an older value. This is a special case of the causal consistency model.
  • Session consistency. This is a practical version of the previous model, where a process accesses the storage system in the context of a session. As long as the session exists, the system guarantees read-your-writes consistency. If the session terminates because of certain failure scenarios a new session needs to be created, and the guarantees do not overlap the sessions.
  • Monotonic read consistency. If a process has seen a particular value for the object any subsequent accesses will never return any previous values.
  • Monotonic write consistency. In this case the system guarantees to serialize the writes by the same process. Systems that do not guarantee this level of consistency are notoriously hard to program.

What is the consistency model offered by this?

Request Number % Count of Servers = current server index.

As it turned out, it is pretty much resolves into “none”. Because each request may be re-directed into a different server, there is no consistency guarantees that we can make.  That can make reasoning about the system really hard.

Instead, RavenDB choose to go another route, and we use the following formula to calculate which server we will use.

Session Number % Count of Servers = current server index.

By using a session, rather than a request, counter to decide which server we will use when spreading reads, we ensure that all of the requests made within a single session are always going to the same server, ensuring we have session consistency. Within the scope of a single session, we can rely on that consistency, instead of the chaos of no consistency at all.

It is funny, how such a small change can have such profound implication.