Ayende @ Rahien

Refunds available at head office

That No SQL Thing – Key / Value stores – Usages

Now that we have a fairly good understanding of what Key / Value stores are, what are they good for? Well, they are really good if you need to lookup things by key :-)

Storing the user’s session is a common example, or the user’s shopping cart. In fact, per user’s data is probably the most common usage for a key value store. The reasons for that are fairly obvious, I would imagine, we have the user’s id already, and we need a well known data. That make it very cheap to access, modify and store at the end of a request.

Other usages include storing already processed data for quick retrieval, consider things like: Favorite Products, Best Of …, etc. The idea here is to have the application access the key / value store using well known keys for those data items, while background processes will update those periodically. It isn’t quite a cache, because there isn’t really a scenario where the value doesn’t exists, but it is a near thing.

One interesting use case that I run into was to store the product catalog in the key/value store, that allowed pulling a particular product’s details in a single call. The key/value store was also used to track how many units were physically in stock, so it was able to tell if the item was available for purchase or not.

Just remember, if you need to do things more complex than just access a bucket of bits using a key, you probably need to look at something else, and the logical next step in the chain in the Document Database.

Comments

Henning Anderssen
04/08/2010 09:24 AM by
Henning Anderssen

Interesting ideas.

Could I use a combination of Key / Value store and for example document db, where you store user session data, shopping cart etc in the K/V store, and your "domain model" in a document db?

Ayende Rahien
04/08/2010 09:26 AM by
Ayende Rahien

Henning,

Yes, that is possible, but the fun part about most document dbs is that they are already key/value stores :-)

Henning Anderssen
04/08/2010 11:01 AM by
Henning Anderssen

Ahh, awesome.

I'm starting to look into MongoDB now. Looks very promising, although it requires a bit of a brainshift in the way I think about storage.

Demis Bellot
04/08/2010 12:11 PM by
Demis Bellot

I would say that only useful for accessing 'a bucket of bits using a key' is only true for primitive key value stores like Memcached.

There are new NoSQL databases (like Redis and MongoDB) with a lot more advanced features opening up a new suite of use-cases:

At mflow (apart from maintaining intelligent in-memory data views) we are using Redis to store rolling error logs (important for maintaining a fast, combined chronological ordered list of error logs for our load-balanced SOA web services) and as a fast in-memory Message Queue for persisting and load-balancing async web service requests.

There are a number of other use-cases for where it is the best solution, like being the optimal data store for high-performance 'web streaming' real-time notification comet servers, that when used in conjunction with a technology like nodejs.org solves the 10k connection problem.

A good example of advanced software that utilizes Redis as a backend that I just noticed on the twitter-sphere today is:

http://www.ohmstudio.com/
Who provide software allowing you to collaboratively compose music online. Their demo video is impressive.

Mark Rogers
04/08/2010 03:00 PM by
Mark Rogers

When I read about how people mention that they actually used a non-RDMS in production, the story usually starts off hopeful and then descends into regret and despair.

John Farrell
04/08/2010 03:36 PM by
John Farrell

@Mark Rogers

[Citation Needed]

Demis Bellot
04/08/2010 04:08 PM by
Demis Bellot

@Mark Rogers I think like any technology there is a risk to over use it. Devs might think hey here's this shiny new tech that's x times faster than our RDBMS and will solve all of our performance and scalability issues only to realize half-implemented way down the line that they have a suite of new problems and actually need that lovely query language in-built into their RDBMS.

Personally I don't view NoSQL data stores as a replacement technology. Rather I believe they make a good complementary technology and are great when they have in-built support for your Use-Case. As Oren has pointed out most key value stores are a good fit when you want to 'just access a bucket of bits using a key'.

Personally we've had great success at mflow.com and have provided a much better and more responsive user experience as a result of utilizing Redis whenever it made sense to. Most of our web services end up hitting optimized 'in-memory data views' which mirrors data that ultimately resides in our sharded RDBMS databases. If you're interested in checking out the user experience said technology and you live in the UK, you can access our closed-beta and as a bonus 'download any free track of your choice' with my limited invite codes attached to this url (we're kinda like social music service that's kinda like itunes+twitter+spotify):

http://www.mflow.com/download/index/mythz007

Erik van Brakel
04/08/2010 11:21 PM by
Erik van Brakel

Hmmm, if I put two and two together, I'd say that a Key-Value store combined with something like Lucene is a really good option for a lot of things. I'm probably wrong here, but as far as I understand, it works a bit like this right?

  • Lucene walks your data, generating indexes (by key), and allows very quick searches by using these indexes to find the key.

  • A key-value store uses keys, obviously.

So to search for something, you use Lucene to find the keys, and fetch them from your KV store.

So searching shouldn't be a problem like that, right? You're probably going to prove me wrong with the next one in the series about document databases showing a far easier way though ;-)

Demis Bellot
04/09/2010 01:42 AM by
Demis Bellot

@Erik yeah that would be golden, and it looks like Oren's heading down that path with his Divan Db.

The problem is that if Lucene walks your data in a background task than you could potentially have stale data in the Search Index. The ideal solution would be something along the lines of Oren's solution where the Search Index is updated in real-time (or close to real-time i think he quoted 25ms) preferably using queue technology.

Ayende Rahien
04/09/2010 07:12 AM by
Ayende Rahien

Erik,

Yes and no.

You would need some way to update the lucene index, and if you need to do multi key access, that is likely to be expensive.

For things like that, I would rather use a doc db, yes.

Ayende Rahien
04/09/2010 07:13 AM by
Ayende Rahien

Demis,

Raven's does that, yep. It provide two things, immediate background indexing and explicit notification about whatever the indexed results are stale or not.

Ayende Rahien
04/09/2010 07:14 AM by
Ayende Rahien

Rob,

I am going to respond to that post in a separate post

Comments have been closed on this topic.