RavenDB, ACID & BASE
Kelly Sommers has been commenting about transactional model on twitter.
"Fully ACID writes but BASE reads" - Go back to the books. You can't selectively opt out of guarantees as you wish. What can client observe?
And some other things in the same vien. I am going to assume that this is relevant for RavenDB, based on other tweetss from her. And I think that this requires more space than allowed on twitter. The gist of Kelly's argument, as I understand it, is that RavenDB isn't ACID because it does BASE reads.
I am not sure that I am doing the argument justice, and it is mostly pieced together from a whole bunch of tweets, so I would love to see a complete blog post about it. However, I think that at least with regards to RavenDB, there is a major misconception going on here.
RavenDB is an ACID database, period. If you put data in, you can be sure that the data will be there and you can get it out again. Indeed, the data you put in is immediately consistent. There is no scenario in which you can do any sequence of read/write/read/write and not immediately get the last committed state for that server. What I think is confusing people is the fact that we have implemented BASE queries. So let us go back a bit and discuss that.
Internally, RavenDB is structure as a set of components. One of those components in the document store, which is responsible for... storing documents (and a lot more besdie, but that isn't very important right now). The document store is ACID. And it ensure consistency, MVCC, durability, and all that other good stuff.
The document store is also limited in the kind of queries that it support. In effect, it supports only the following:
- Document by key
- Document by key prefix
- Documents by update order
When talking about ACID, I am talking about this part of the system, in which you are writing documents and loading them by their id or by update order.
Note that in RavenDB, we use the term "load" to refer to accessing the document store. That is probably the cause for confusion. Loading is never inconsistent and obeys a strict snapshot isolation interpretation of the data. That means that asking something like: "give me users/123" is always going to give you the immediately consistent result.
In constrast to that, RavenDB also have the index store. The index store allows us to create complex queries, and that is subject to eventual consistency. That is by design, and a large part of what makes RavenDB able to dynamically adjust its own behavior at runtime based on production usage.
We have a separate background processes that apply the indexes to the incoming data, and write them to the actual index store. This is done in direct contrast to other indexing systems (for example, RDBMS) in which indexes are updated as part of the same transaction. That leads to several interesting results.
Indexes aren't as expensive - In RDBMS, the more indexes you have, the slower your writes become. In RavenDB, indexes have no impact on the write code path,. That means that you can have a lot more indexes without worrying about it too much. Indexes still have cost, of course, but in most system, you just don't feel it.
Lock free reads & writes - Unlike other databases, where you have to pay the indexing cost either on write (common) or read (rare), with RavenDB both read & writes operate without having to deal with the indexing cost. When you make a query, we will immediately give you an answer from the results that we have right now. When you perform a write, we will process that without waiting for indexing.
Dynamic load balancing - because we don't promise immediate consistency, we can detect and scale back our indexing costs in time of high usage. That means that we can shift resources from indexing to answering queries or accepting writes. And we will be able to pick up the slack when the peek relaxes.
Choice - one of the things that we absolutely ensure is that we will tell you when you make a query and we give you results that might not be up to date. That means that when you get the reply, you can make an informed decision. "This query is the result of all the data in the index as of 5 seconds ago" - decide if this is good enough for you or if you want to wait. Unlike other systems, we don't force you to accept whatever consistency model you have. You get to choose if you want to answers I can give you right now, or if you want to wait until we can give you the conistent version, that is up to you.
Smarter indexes - under the hood, RavenDB uses lucene for indexes. Even without counting things like full text or spatial searches, we get a lot more than what you get from the type of indexes you are used to in other systems.
For example, from the MongoDB docs (http://docs.mongodb.org/manual/core/index-compound/):
The following index can support both these sort operations:
However, the above index cannot support sorting by ascending username values and then by ascending date values, such as the following:
In constrant, in RavenDB, it is sufficent that you just tell us what you want to index, and you don't need to worry about creating two indexes (and paying twice the price) if you want to allow users to sort the grid in multiple directions.
On the fly optimiaztions - the decision to separate the document store and the index store to separate components has paid off in ways that we didn't realy predict. Because we have the sepration, and because indexes are background processes that don't really impact the behavior of the rest of the system (outside of resources consumed), we have the freedom to do some interesting things. One of them is live index rebuild.
That means that you can create an index in RavenDB, and the system would go ahead and create it. At the same time, all other operations will go on normally. It can be writes to the document store, it can be updates to the other indexes. Anything at all.
Just to give you some idea about this feature, you'll note that this feature exists only in the Enterprise edition for SQL Server, for example.
And the implications of that are that we can actually create an index in production. In fact, we do so routinely, that is the basis of our ad hoc querying ability. We analyze the query, the query optimizer decides where to dispatch it, and will create the new index if that is needed. There is a whole host of behavior just in this one feature (optimizations, heuroistics, scale back, etc) that we can do, but for the most part, you don't care. You make queries to the server. RavenDB analyze them and product the most efficent indexes to match you actual production load
.Anyway, I am starting to go on about RavenDB, and that is rarely short. So I'll summarize. RavenDB is composed of two distinct parts, the ACID document store, which allow absolute immediate consistency for both reads & writes. And it also have the index store, which is eventually consistent, updated in the background and are always available for queries. They'll also do zero waits (unless you explicitly desire this) for you and give you a much better and more consistent performance.
Oh, and a final number to conclude. The average time between a document update in the document store and the results showing up in the index store? That is 23ms*.
(And I blame 15.6ms of that on the clock resolution).
Let's refer to a well known database researcher (and NewSQL DB developer) to do the talking: http://hackingdistributed.com/2013/03/23/consistency-alphabet-soup/ and: http://hackingdistributed.com/2013/02/14/whats-nosql/
Let's refer to another well known database researcher (and also NewSQL DB developer) to chime in as well: http://dbmsmusings.blogspot.nl/2012/05/if-all-these-new-dbms-technologies-are.html
In short: you may say RavenDB is 'ACID compliant', but that's only true for your definition of 'ACID'. Nothing wrong with that though, there's no free lunch. If you want to have a much faster, distributed system compared to a normal ACID compliant RDBMS system, you have to make a trade off somewhere. That's been described some time ago here, by one of the most well-known database researchers to day: http://cacm.acm.org/blogs/blog-cacm/50678-the-nosql-discussion-has-nothing-to-do-with-sql/fulltext
But if the Document Store is ACID and the Index Store is BASE, does that make Raven ACID or BASE as a product?
I feel like this is a bit of a murky area because it depends on the usage (If you use both the document Store and Index Store, then it's neither BASE nor ACID, although it is more BASE than ACID)
P.S. Maybe Raven is BASID (Basically Available, Soft State, Isolated, Durable) ? (Note the missing Atomic and Consistent, as updates to indexes are neither Atomic or Consistent with Document updates)
Just as a note you can change windows clock resolution if you want to be 1ms http://msdn.microsoft.com/en-us/library/dd757624%28VS.85%29.aspx
I put this on twitter but transactions are not generally really interesting until you have more than one operation to make atomic. As you are generally dealing with single updates and an eventually consistent index it's not really interesting.
It's perfectly valid as well to have different strategies for different operations... Btw you could even make the lucene stuff fully consistent if you marked where the transaction log at beginning of query and scanned forward for matching records on the read.
As a developer, you've explained perfectly what parts of RavenDB I have to think about when designing an application and the trade offs of the model you have chosen. So I'm not worried so much about which acronym best describes RavenDB. I have one for you AWESOME! (A Well Engineered Storage Option Made Easy).
If people have a problem with how Raven behaves according to a definition then maybe you should build a feature that is called "Tripping On ACID", where when enabled Indexes are updated with every commit. This would conform to the rule, but would suck for performance.
I respect Kelly Sommers, she is a very intelligent technologist and she's made my head hurt on multiple occasions and I always enjoy reading her blog posts.
Have to agree with Kelly - you can't be 'partially ACID'. And that's nice about it. Strict rules, no randomness or timing dependencies. BASE acronym is somewhat unjust in case of RavenDB, but it's a wide area where you can do all kinds of compromises. And you did, deliberately. Why pretend nothing was sacrified?
Asking the user to hit 'F5' because the next screen did not have his recently added whatever is not much of a sacrifice.
There's one interface for writing (IDocumentSession, typically you Load a document because you are going to mutate it) and another interface for Querying (IQueryable of T). There's gotta be a name for this...
A small caveat to keep in mind. If your query returns documents, as opposed to projections from the index, those documents are fully consistent as well.
On the plus side, you have the latest. On the "downside", the document may not, now, match the query parameters.
How much of a problem this is presents depends on use cases.
@João Bragança - You can easily avoid that problem by waiting for the etag of the item you just wrote to be included in the query. I use this technique all the time. See http://stackoverflow.com/a/18815765/634824
Regarding Calvin transactions (http://dbmsmusings.blogspot.nl/2012/05/if-all-these-new-dbms-technologies-are.html):
"You're right that Calvin doesn't automatically promise read-after consistency. However, each committed transaction returns to the client its logical timestamp. This timestamp is actually the transaction's place in the serial transaction ordering to which all replicas' executions must be equivalent. A client could therefore specify that a read must see a version at least as new as the timestamps of previous writes. It would be straightforward to augment Calvin's client-side interface to track the history of the client's database interactions and automatically added these annotations to read requests, so that they could safely be sent to any replica."
This is how RavenDB works for queries on indexing if you want to wait for your writes.
And I think that the guy on the last link is missing something important. Not being constrained to the relational model means that you can do a lot more.
Catalin, That depend on what operation you are doing. You can choose for each operation how to behave.
Gregg, This is really something that you shouldn't do, though. Changing the timer resolution increased the power usage of the system considerably. Also, RavenDB transactions are not limited to a single operation. You can do multiple operations (indeed, that is frequently the case) which are covered under that transaction.
Khalid, That is a cool acronym. Can we use that?
And Kelly is pretty great techie, yep. I very much enjoy conversations with her, even/especially when we don't see eye to eye.
Rafal, If you want complete ACIDity, you use the API that ensures that, simple. There are additional things that you can do, which have different guarantees.
If you think Stonebraker misses something in the Database field and especially something 'important', you should do your homework. https://en.wikipedia.org/wiki/Michael_Stonebraker
Like someone once said: what Stonebraker doesn't know about databases fits on a small stamp.
What I don't get is why you try so hard to prove that RavenDB can do everything an RDBMS can do and without the constraints that come with all that, without any trade off, while at the same time you didn't make sure you can actually do all that like Hyperdex Warp can or VoltDB? It's perfectly fine not to be ACID compliant, just say you are. The thing now is that you tell people your database IS ACID compliant and people will believe that, while at the same time it actually isn't.
Nice blog post, I've been wondering about the whole "is RavenDB ACID things, especially after seeing Kelly's tweets.
This made me laugh though:
"and it is mostly pieced together from a whole bunch of twits"
I think you means "bunch of tweets". Twits means something a bit different, at least in the UK, http://en.wikipedia.org/wiki/The_Twits
Frans, I never said once that RavenDB can do everything that an RDBMS can do without the constraints. We made a different design choices which allows us to do things _differently_. Often, that means that we can do things better for the user. For example, live index rebuilds, or adjusting resource usage according to load.
And I don't really think that I could be clearer regarding ACID.
Matt, Thanks, updated the tweets.
of course you can use it, why do you even have to ask :)
Khalid, It is polite.
Sure, it's a lot like the user hitting F5 until he sees the result he expects :)
Although most of the time this is unnecessary. The document you just worked on? You still have it in memory. Why query for it again?
@Frans if those people say RavenDB is not ACID, they're wrong. People are always allowed to be wrong.
I made this exact statement to Sommers.
You claim that, because of document store and the index store separation, you have live index rebuild which is only available in Enterprise edition for SQL Server.
"That means that you can create an index in RavenDB, and the system would go ahead and create it. At the same time, all other operations will go on normally".
What you're talking about is index creation (not live rebuild). You can create new index in Sql Server and every operation will also function normally. Also, when you are doing what you call "live rebuild" you are getting indices in consistent state but they are always consistent in Sql Server. Live rebuild is only used to remove fragmentation and reclaim disk space which is something totally different.
Marka, See here: http://technet.microsoft.com/en-us/library/ms177442.aspx
"This topic describes how to create, rebuild, or drop indexes online in SQL Server 2012 by using SQL Server Management Studio or Transact-SQL. The ONLINE option allows concurrent user access to the underlying table or clustered index data and any associated nonclustered indexes during these index operations. For example, while a clustered index is being rebuilt by one user, that user and others can continue to update and query the underlying data. When you perform data definition language (DDL) operations offline, such as building or rebuilding a clustered index; these operations hold exclusive locks on the underlying data and associated indexes. This prevents modifications and queries to the underlying data until the index operation is complete."
That is what I meant.
My point is - what you're talking about is not live index rebuild. When you "rebuild" index, do you recreate index for ALL data from beginning or you just update index based on recent modifications (I beleave doing first option would be an overkill)? Index rebuild in Sql Server RECREATES index in order to remove fragmentation and compact disk space. You can also remove fragmentation with index reorganize which can be done online and comes in Standard Edtition.
Marka, What I am talking about here is the ability to create an index on the fly, without stopping any other operations. It can be index rest (what you call rebuild), it can be creating / deleting another index.
It's seams you're right. I was sure that index creation IS ONLINE operation, but apparently it isn't (At least without Enterpise edition).