When you pit RavenDB & SQL Server against one another…
Here is how it works. I hate benchmarks, because they are very easily manipulated. Whenever I am testing performance stuff, I am posting numbers, but they are usually in reference to themselves (showing improvements).
That said…
Mark Rodseth .Net Technical Architect at Fortune Cookie in London, UK and he did a really interesting comparison between RavenDB & SQL Server. I feel good about posting this because Mark is a totally foreign agent (hm…. well, maybe not that ) but he has no association with RavenDB or Hibernating Rhinos.
Also, this post really made my day.
Update: Mark posted more details on his test case.
Mark setup a load test for two identical applications, one using RavenDB, the other one using SQL Server. The results:
SQL Load Test
Transactions: 111,014 (Transaction = Single Get Request)
Failures: 110,286 (Any 500 or timeout)
And for RavenDB ?
RavenDB Load Test
Transactions: 145,554 (Transaction = Single Get Request)
Failures: 0 (Any 500 or timeout)
And now that is pretty cool.
Comments
I may have missed a source link, and if that is the case, I apologize.
However, I'm not sure how this test shows anything of value. You start out with a quote in your post which seems extremely fitting in this example. We get absolute numbers, but have no information on the actual test being performed. We can't see the complete data model. We can't see the source. We can't see the caching/setup involved.
Given that just a single request is being made, this seems to indicate just a "display" page being rendered, in which case RavenDB is by far the best - as we can aggregate all the necessary data. No doubt there. That does however ignore all other potential parts of the system.
I'd really like to see the source and setup so we can evaluate the results we're seeing. Until then, how should we interpret the results?
Mark, 4th paragraph, is where I mention the source: http://tech-rash.blogspot.com/2012/02/is-raven-db-all-its-cracked-up-to-be.html
Nice..great job to you and your team! Assume us Java folks can only use it over the REST API.
Sony, Yes, this is available through REST to java.
Show code. Without code this is just a puctures.
Ayende, I saw the post and read it, and that's actually what I'm referring to. In Marks post, I see no mention of the actual test or source being run. I'll comment on there as well and ask for it, for until then, I think the test is of limited value.
You, better than any, should know they're "very easily manipulated".
I'd imagine the SQL query would have been slower than RavenDB due to the number of joins required to display the page.
Under load (1000 concurrent users) you'd see the usual issue of using up all the ASP.NET request threads whilst waiting for each query to finish.
Would be interested to see if they'd used async page handlers and async ADO.NET to avoid this issue, and then give some actual performance metrics on average request length and number of requests per second. Instead we are looking at failed requests which are likely just due to threadpool starvation, rather than comparing the real performance characteristics of the underlying databases.
Raven's results are great and encouraging, but I would not underestimate SQL server - sometimes it's just astonishing how fast it can be if we don'd stand in its way. For example, you were happy about RavenDB being able to load 1000 Json documents per second. Few months ago I did a test of loading Json data into SQL server table and it was able to swallow 15 000 documents per second, one by one, transactional, without any batching (with batching it was a little faster - 17000 per second). Everything depends on the choice of a test case.
The benchmark is so out of whack that it likely means nothing. 90% of SQL requests failing... ;-) At the very least this was not a test for throughput. This is not the kind of benchmark I would want to promote my product with because it is obviously bogus. Is the source available?
A 99% failure rate with SQL Server? With only 61 requests per second? There is no way I can believe that was anything more than a badly designed test. And the fact that you are citing it actually lessens my opinion of RavenDB.
Oh actually there were 99% failures for SQL server! lol what an embarrassing benchmark. I'd have dialed back the load so that virtually no query was failing. That would have resulted in a comparable number.
The fact that Raven is faster is indisputable and was clear without starting the benchmark. What is important is how much faster!
I think that the graphs should've been truncated at the point when 250+ concurrent requests caused SQL server transactions to fail. If you just compare the graphs before that point it's showing that Raven's throughput is magnitudes higher.
What does "bytes received" mean? It could mean higher overhead for RavenDB. I think there is actually no throughput data, measured in equivalent transactions per sencond, there at all!
Hmm... on second look I realize I was incorrectly assuming that "bytes received" meant the amount of data returned to the caller of the DB query per second.
You're right, these charts make no sense at all. The way I'm reading it now says "for X number of concurrent transactions with equivalent queries, RavenDB is returning magnitudes more bytes than SQL server", which really can't be considered a good quality so either I'm confused or these charts make no sense.
Pretty amazing I must say... too bad people don't read the source (at least not the whole article).
I don't think the testing result is correct completely. What are the version of databases in testing? what are the configuration of database servers?
It would be much more useful if one could read the text on the images.
LOL :) A database 'benchmark' done on a VM. Databases are I/O bound in general, testing 'performance' on a VM is stupid: the VM's I/O subsystem is unpredictable and as I/O is a bottleneck in databases, it's a significant factor. That the charts are totally bonkers is caused IMHO mainly by the usage of a VM.
But what surprises me more is that Oren posts this, considering he wrote posts about ormbattle.net and NHibernate and how skewed the results were (they are btw, that's the fun part here).
Oren, you of all people should know: the less features you have, the faster you are in database/data-access benchmarks. To illustrate the point, a simple key-value dictionary serialization system is faster than your DB will ever be, simply because it can win in the area BEFORE the I/O bottleneck hits, namely it only has to serialize data and that's it.
I'm sure you'll disagree with me, that's fine. I'm not here to defend MS' database system, they have a large army of MVPs to do that for them. I think you should look at what 'Jonathan Allen' said above: the benchmark you posted is nothing better than what the fine folks over at ormbattle.net try to do.
Frans,
VM aside, if you look at the actual SQL being executed, you will immediately see why it failed so much. There is essentially one large table with no indexes other than the PK. The SPROC performing the query tells the rest of the story. There are not 1, but 3 (sometimes 4) temp tables created. There is a loop performing highly procedural logic which queries from real tables (including the large table containing no indexes), and inserts into the temp tables. Finally it does some seriously heavy manipulation of the data in the temp tables before returning a result. With a large enough data set, I wouldn't be surprised if this takes 10s of seconds or even minutes to execute on even fastest hardware.
In this particular test, SQL Server stood no chance. I think with a proper model, and proper query, and tuning, SQL Server is likely to outperform RavenDB any day. But it's always going to be Apples-to-Oranges comparison. And Raven's programming model alone may be worth the compromise of a few ms of query time.
This test is nothing but just a piece of crap, just look at the EAV schema used on the SQL side
+2 Frans I remember those posts (and your responses)… classic
-1 Ayende Come on Ayende… nobody minds self promotion but this is a little much
Ouch - come on...
That's the worst "benchmark" I've ever seen. Basically he's saying we had this horrible mess someone created on SQL Server. We Can't change it. But, we're going to write a new implementation on RavenDB which doesn't have the horrible mess and then compare the two on performance.
You're usually so detailed oriented, but I have to guess that you didn't read the entire write-up or the follow-ups - they are at:
The follow-up: http://tech-rash.blogspot.com/2012/02/ravendb-vs-sql-follow-up.html
The horrible mess they created on SQL Server: https://docs.google.com/document/d/1h4kiNE_g-jrxpNIUrwMjYRSonLa0XOTqRq9C38g7dI0/edit?pli=1
It would be great to see some actual performance comparisons with some basic inserts, updates and deletes to get a sense for how Raven performs vs SQL Server.
Comment preview