Ayende @ Rahien

Refunds available at head office

When you pit RavenDB & SQL Server against one another…

Here is how it works. I hate benchmarks, because they are very easily manipulated. Whenever I am testing performance stuff, I am posting numbers, but they are usually in reference to themselves (showing improvements).

That said…

Mark Rodseth .Net Technical Architect at Fortune Cookie in London, UK and he did a really interesting comparison between RavenDB & SQL Server. I feel good about posting this because Mark is a totally foreign agent (hm…. well, maybe not that Smile ) but he has no association with RavenDB or Hibernating Rhinos.

Also, this post really made my day.

Update: Mark posted more details on his test case.

Mark setup a load test for two identical applications, one using RavenDB, the other one using SQL Server. The results:

SQL Load Test
Transactions: 111,014 (Transaction = Single Get Request)
Failures: 110,286 (Any 500 or timeout)

And for RavenDB ?

RavenDB Load Test
Transactions: 145,554 (Transaction = Single Get Request)
Failures: 0 (Any 500 or timeout)

And now that is pretty cool.

Comments

Mark S. Rasmussen
02/09/2012 06:34 PM by
Mark S. Rasmussen

I may have missed a source link, and if that is the case, I apologize.

However, I'm not sure how this test shows anything of value. You start out with a quote in your post which seems extremely fitting in this example. We get absolute numbers, but have no information on the actual test being performed. We can't see the complete data model. We can't see the source. We can't see the caching/setup involved.

Given that just a single request is being made, this seems to indicate just a "display" page being rendered, in which case RavenDB is by far the best - as we can aggregate all the necessary data. No doubt there. That does however ignore all other potential parts of the system.

I'd really like to see the source and setup so we can evaluate the results we're seeing. Until then, how should we interpret the results?

Ayende Rahien
02/09/2012 06:45 PM by
Ayende Rahien

Mark, 4th paragraph, is where I mention the source: http://tech-rash.blogspot.com/2012/02/is-raven-db-all-its-cracked-up-to-be.html

Sony Mathew
02/09/2012 06:45 PM by
Sony Mathew

Nice..great job to you and your team! Assume us Java folks can only use it over the REST API.

Ayende Rahien
02/09/2012 06:46 PM by
Ayende Rahien

Sony, Yes, this is available through REST to java.

gandjustas
02/09/2012 06:50 PM by
gandjustas

Show code. Without code this is just a puctures.

Mark S. Rasmussen
02/09/2012 06:54 PM by
Mark S. Rasmussen

Ayende, I saw the post and read it, and that's actually what I'm referring to. In Marks post, I see no mention of the actual test or source being run. I'll comment on there as well and ask for it, for until then, I think the test is of limited value.

You, better than any, should know they're "very easily manipulated".

Neil Mosafi
02/09/2012 06:54 PM by
Neil Mosafi

I'd imagine the SQL query would have been slower than RavenDB due to the number of joins required to display the page.

Under load (1000 concurrent users) you'd see the usual issue of using up all the ASP.NET request threads whilst waiting for each query to finish.

Would be interested to see if they'd used async page handlers and async ADO.NET to avoid this issue, and then give some actual performance metrics on average request length and number of requests per second. Instead we are looking at failed requests which are likely just due to threadpool starvation, rather than comparing the real performance characteristics of the underlying databases.

Rafal
02/09/2012 07:02 PM by
Rafal

Raven's results are great and encouraging, but I would not underestimate SQL server - sometimes it's just astonishing how fast it can be if we don'd stand in its way. For example, you were happy about RavenDB being able to load 1000 Json documents per second. Few months ago I did a test of loading Json data into SQL server table and it was able to swallow 15 000 documents per second, one by one, transactional, without any batching (with batching it was a little faster - 17000 per second). Everything depends on the choice of a test case.

tobi
02/09/2012 07:13 PM by
tobi

The benchmark is so out of whack that it likely means nothing. 90% of SQL requests failing... ;-) At the very least this was not a test for throughput. This is not the kind of benchmark I would want to promote my product with because it is obviously bogus. Is the source available?

Jonathan Allen
02/09/2012 07:13 PM by
Jonathan Allen

A 99% failure rate with SQL Server? With only 61 requests per second? There is no way I can believe that was anything more than a badly designed test. And the fact that you are citing it actually lessens my opinion of RavenDB.

tobi
02/09/2012 07:15 PM by
tobi

Oh actually there were 99% failures for SQL server! lol what an embarrassing benchmark. I'd have dialed back the load so that virtually no query was failing. That would have resulted in a comparable number.

The fact that Raven is faster is indisputable and was clear without starting the benchmark. What is important is how much faster!

Dan
02/09/2012 10:48 PM by
Dan

I think that the graphs should've been truncated at the point when 250+ concurrent requests caused SQL server transactions to fail. If you just compare the graphs before that point it's showing that Raven's throughput is magnitudes higher.

tobi
02/09/2012 10:52 PM by
tobi

What does "bytes received" mean? It could mean higher overhead for RavenDB. I think there is actually no throughput data, measured in equivalent transactions per sencond, there at all!

Dan
02/09/2012 11:06 PM by
Dan

Hmm... on second look I realize I was incorrectly assuming that "bytes received" meant the amount of data returned to the caller of the DB query per second.

You're right, these charts make no sense at all. The way I'm reading it now says "for X number of concurrent transactions with equivalent queries, RavenDB is returning magnitudes more bytes than SQL server", which really can't be considered a good quality so either I'm confused or these charts make no sense.

Remco Ros
02/09/2012 11:45 PM by
Remco Ros

Pretty amazing I must say... too bad people don't read the source (at least not the whole article).

Hung Tran
02/10/2012 02:54 AM by
Hung Tran

I don't think the testing result is correct completely. What are the version of databases in testing? what are the configuration of database servers?

ThomasW
02/10/2012 07:31 AM by
ThomasW

It would be much more useful if one could read the text on the images.

Frans Bouma
02/10/2012 11:21 AM by
Frans Bouma

LOL :) A database 'benchmark' done on a VM. Databases are I/O bound in general, testing 'performance' on a VM is stupid: the VM's I/O subsystem is unpredictable and as I/O is a bottleneck in databases, it's a significant factor. That the charts are totally bonkers is caused IMHO mainly by the usage of a VM.

But what surprises me more is that Oren posts this, considering he wrote posts about ormbattle.net and NHibernate and how skewed the results were (they are btw, that's the fun part here).

Oren, you of all people should know: the less features you have, the faster you are in database/data-access benchmarks. To illustrate the point, a simple key-value dictionary serialization system is faster than your DB will ever be, simply because it can win in the area BEFORE the I/O bottleneck hits, namely it only has to serialize data and that's it.

I'm sure you'll disagree with me, that's fine. I'm not here to defend MS' database system, they have a large army of MVPs to do that for them. I think you should look at what 'Jonathan Allen' said above: the benchmark you posted is nothing better than what the fine folks over at ormbattle.net try to do.

Joseph Daigle
02/10/2012 01:26 PM by
Joseph Daigle

Frans,

VM aside, if you look at the actual SQL being executed, you will immediately see why it failed so much. There is essentially one large table with no indexes other than the PK. The SPROC performing the query tells the rest of the story. There are not 1, but 3 (sometimes 4) temp tables created. There is a loop performing highly procedural logic which queries from real tables (including the large table containing no indexes), and inserts into the temp tables. Finally it does some seriously heavy manipulation of the data in the temp tables before returning a result. With a large enough data set, I wouldn't be surprised if this takes 10s of seconds or even minutes to execute on even fastest hardware.

In this particular test, SQL Server stood no chance. I think with a proper model, and proper query, and tuning, SQL Server is likely to outperform RavenDB any day. But it's always going to be Apples-to-Oranges comparison. And Raven's programming model alone may be worth the compromise of a few ms of query time.

Paul
02/10/2012 09:30 PM by
Paul

This test is nothing but just a piece of crap, just look at the EAV schema used on the SQL side

meisinger
02/11/2012 03:41 AM by
meisinger

+2 Frans I remember those posts (and your responses)… classic

-1 Ayende Come on Ayende… nobody minds self promotion but this is a little much

Mike
02/23/2012 08:10 AM by
Mike

Ouch - come on...

That's the worst "benchmark" I've ever seen. Basically he's saying we had this horrible mess someone created on SQL Server. We Can't change it. But, we're going to write a new implementation on RavenDB which doesn't have the horrible mess and then compare the two on performance.

You're usually so detailed oriented, but I have to guess that you didn't read the entire write-up or the follow-ups - they are at:

The follow-up: http://tech-rash.blogspot.com/2012/02/ravendb-vs-sql-follow-up.html

The horrible mess they created on SQL Server: https://docs.google.com/document/d/1h4kiNE_g-jrxpNIUrwMjYRSonLa0XOTqRq9C38g7dI0/edit?pli=1

It would be great to see some actual performance comparisons with some basic inserts, updates and deletes to get a sense for how Raven performs vs SQL Server.

Comments have been closed on this topic.