Ayende @ Rahien

Ayende Rahien commented on Benchmarks are useless, yes, again

Wed, 19 Aug 2009 05:01:49 GMT

@All, This thread is getting repetitive, and I don't see any value in continuing the same discussion over & over again. Comments are closed.

Ayende Rahien commented on Benchmarks are useless, yes, again

Wed, 19 Aug 2009 04:59:52 GMT

Alex, Because what you are publishing is not tomatoes calories value, you are publishing the nutrient values in the water used to grow a different set tomatoes.

Alex Yakunin commented on Benchmarks are useless, yes, again

Wed, 19 Aug 2009 04:38:26 GMT

Ok, my remark about "real life performance" in your "swimming in Sahara" style, if you want: If I buy tomatoes, I'm interested in their calorie content, but not in calorie content of some soup with tomatoes. But you're suggesting all tomato producers must publish calorie content of some soup with tomatoes. Why? Just because this will make difference between your fat and our slim ones less visible.

Alex Yakunin commented on Benchmarks are useless, yes, again

Wed, 19 Aug 2009 04:22:02 GMT

Oren, I answered to this many, many times. Say something new, please ;) You provided zero facts & evidences. Comparisons like "swimming competition in the Sahara" may work on public conversation, but never works @ forums & blogs.

Ayende Rahien commented on Benchmarks are useless, yes, again

Wed, 19 Aug 2009 04:03:41 GMT

Alex, Your materialization tests has as much relation to the real world as a swimming competition in the Sahara. And I think you are missing the point with reducing the # of queries. You are forcing thousands of queries and then try to optimize that. Don't start with thousands of queries, your life will be better

Alex Yakunin commented on Benchmarks are useless, yes, again

Wed, 19 Aug 2009 03:42:10 GMT

> imho speed optimizations are directly connected to the whole code perfection. That's what we were talking about on the first page of ORMBattle.NET. Not absolutely true, but in wide majority of cases - yes. Or think about LINQ at least. And, guys... I just thought it is simply can't be true you're saying ORM performance isn't important. And asked the Google. "Ayende NHibernate performance" - first link: [ayende.com/.../NHibernatePerformanceConcerns.aspx](http://ayende.com/Blog/archive/2006/10/18/NHibernatePerformanceConcerns.aspx) Few quotes: - "The first problem that Darrel has is with the memory consumtion. I will start right off by saying that I have been a devot user of NHibernate*for over two years now, and I'm building big, complex systems using it. Memory consumtion was never an issue with NHibernate." - think about our materialization test, which results, as I wrote, are mainly related to memory consumption. - "Optimizing NHibernate's performance is almost solely focused on reducing the amount of queries ..." CUD batches are a good example of reducing amount of queries, yes? I'm almost sure, Oren, I can find at least 10 more articles in your blog related to NH performance. I just don't want to spend more time on this. And in the end of all: "nhibernate performance" = 0.5M results in Google. That's definitely not important.

houl commented on Benchmarks are useless, yes, again

Tue, 18 Aug 2009 17:49:51 GMT

And by the way - the more speed reserve you have - the more complicated/useful features you can implement. I think everybody will agree that for example real-time full speech recognition would be rather easy task if we had 10^12 times more calculation power and RAM space in usual PC :) So it is also a good strategy - gain the same solution which works twice faster, and then "spend" its surplus speed for additional features, which will make this product more attractive than any other :)

houl commented on Benchmarks are useless, yes, again

Tue, 18 Aug 2009 17:38:10 GMT

Ayende: "All the numbers you throw around are _maybe_ accurate for lab situations." Atmosphere in server rooms sometimes much closer to labs, not to real life ;)

houl commented on Benchmarks are useless, yes, again

Tue, 18 Aug 2009 17:34:15 GMT

Frans: "I mean, I can't imagine you're doing all this to give telerik or the EF team more attention... " sometimes russians can be absolutely unpredictable in their generosity :-D

houl commented on Benchmarks are useless, yes, again

Tue, 18 Aug 2009 17:29:59 GMT

Frans: "what's the point of throwing time at this" i can suppose at least three variants: 1) it is just interesting and has educational value - if something in other ORM is faster - then there is obvious possibility for improvements in this "something" - so speed comparisons allow to reveal points of interests in improvements of algorithms/solutions 2) DO team can "kill three rabbits with one shot" - create optimal benchmark suite for ORMs, have permanent public attention on them, and always know current situation with ORMs speed comparing them to DO - so if there any niche of speed-critical projects for ORM will appear in future - DO will become monopolistic in a moment in that niche without any concurents - if all of you will continue think in the way you've demonstrated here about no need of high-speed materialization etc :) 3) imho speed optimizations are directly connected to the whole code perfection. "Fully optimized" and "Perfect" - are almost synonims - including code style, transparency for understanding, bugs absence, etc. So the company that have fastest solution can be almost 100% sure that it really has best product - if features list is the same of course. Now look at #2 - about "compare others with their own and permanently maintain it to be fastest.

Frans Bouma commented on Benchmarks are useless, yes, again

Tue, 18 Aug 2009 15:41:59 GMT

Heh, leave it to the puzzle boys to nail the point home :D @Alex, What I wondered today was: now that DO isn't mentioned in the results, what's the incentive for the site now? It was a marketing campaign, but as DO isn't in the list of results anymore (and visitors will only peek at the green/red squares), what's the point of throwing time at this (== money) at all? Or are you re-adding DO in 2 weeks from now? I mean, I can't imagine you're doing all this to give telerik or the EF team more attention... @Mats, glad you reanimated your blog ;) Interesting read indeed.

Alex Yakunin commented on Benchmarks are useless, yes, again

Tue, 18 Aug 2009 15:30:40 GMT

> Materialization and Query Performance is NOT the problem with O/R mapping. > Let me just make completely sure that you _do_ agree: you do agree, then, that until you actually complement with a test that shows the mapper in context of a real application doing something real, it is impossible to know the actual importance of the numbers from your benchmark? Yes, I understand we can prove this only by other tests. Theory is just theory. Ok, any ideas on relatively simple, but "real life" tests except TPC-X (likely, we must implement them anyway) are welcome. > The problems are: Agree with all. Frankly speaking, misusing of a particular ORM is something like normal... And I agree, NH is good from this point: there is a huge community. > IMO, you are just throwing development money away when you optimize for materialization Let's see... I like to develop efficient things. As well as show them :)

Ayende Rahien commented on Benchmarks are useless, yes, again

Tue, 18 Aug 2009 12:52:01 GMT

Alex, All the numbers you throw around are _maybe_ accurate for lab situations. In practice, the numbers are tens of milliseconds.

Roger Alsing commented on Benchmarks are useless, yes, again

Tue, 18 Aug 2009 12:39:49 GMT

@Alex,I'm sorry but you are solving the wrong problem... Materialization and Query Performance is NOT the problem with O/R mapping. The problems are: It's hard to create mappings (e.g. in XML). Alot of people mess up with Lazy Load and casue ripple load havoc in their systems. Error messages are often hard to understand since the inner workings of a mapper is quite complex. The community around NH solves some of those problems. e.g. Fluent NH or Castle AR (which by the way has the best and most descriptive errormessages of all frameworks that I have ever seen) IMO, you are just throwing development money away when you optimize for materialization / query performance since the mapper performance is pretty much dwarfed by all other factors in a big app (as already stated by others). its a quite neat PR ploy tough.. //Roger

Mats Helander commented on Benchmarks are useless, yes, again

Tue, 18 Aug 2009 12:37:44 GMT

>slightly bitowards should be: slightly biased towards

Mats Helander commented on Benchmarks are useless, yes, again

Tue, 18 Aug 2009 12:33:22 GMT

You DO have a good and important point in your observation that as network latency and RAM sizes improve (and disc access times improve as well, for that matter), then we will see less resources drained here, and as a consequence the percentage swallowed by the mapper will be larger. But what exactly are those percentages, now and in the future? Without tests, we just don't know. With all due respect, just doing the maths doesn't tell enough - perhaps you have done real stress testing on big iron stuff, and then you already know this, but if not - prepared to be surprised when comparing to your math based projections ;-) In fact, it does seem to me that some of your arguments/math examples are slightly bitowards _inflating_ the importance of the mapper (when testing only data access, increasing the data volumes may seem like you are making the test more realistic, but it also serves to cement an application abstraction where the application apparently only does data access, which is very misleading) So I still say actual tests - Pet Shop style, preferrably (see my new blog post at [matshelander.blogspot.com/.../...ti-benchmark.html](http://matshelander.blogspot.com/2009/08/pet-shop-anti-benchmark.html) ) are needed to put any of these numbers into any kind of useful perspective. So, finally: >> You do agree with my logic, no? >Yes, I agree with this. Let me just make completely sure that you _do_ agree: you do agree, then, that until you actually complement with a test that shows the mapper in context of a real application doing something real, it is impossible to know the actual importance of the numbers from your benchmark? Since, again, if the overall overhead is low enough (say 5%) then any performance benefit is likely pointless (until future breakthroughs in physics that will for some reason be strangely constrained to applying to t´he network and RAM but not to CPU speed (?) which may completely change the equations). I also wonder if you agree that math-based estimates and extrapolations (such as you have provided so far), even if the math itself is decent, is hardly a substitute for a real test? I realize that not yet having had the time to do the tests I ask, you try to provide the next best thing, which is a reasonable estimate. My problem with this is that the whole point of the benchmark you have created (and currently are using in a marketing drive it seems) may in fact stand or fall with the tests I suggest. So the best thing would perhaps have been to not publish at all until you had the tests to put the numbers into context, but imo the next best thing is then NOT to come with estimates, but to confess that currently the relevance of your numbers is entirely unknown and you will have to return when you have actually run some tests. I hope I don't sound harsh, but you did mention arguing is a sport for you - well, it is for me too ;-) /Mats

Mats Helander commented on Benchmarks are useless, yes, again

Tue, 18 Aug 2009 12:32:41 GMT

>It seems I mentioned all show stoppers<
Well, the database does work, even if it is in RAM - faster than touching disk, of course, but still work. Incidentally, this work can be considered part of the actual "application logic", which if used as an overreaching term (as contrasted to just running a loop) I would say can be used to cover the bulk of the showstoppery you left out. But you do raise an interesting point with regards to the network throughput and latency (and an interesting fact - I have to say "wow" to that 2.5 microsec latency!! :-O) "If you're caring about performance, 10Gbit channels are right for you" That is just the point - do you care that much about performance? Do you care _forever_ about improving performance or will there be some point in a given application where performance is good enough and other aspects become more important? What I propose is that for a lot of people, the following applies: 1) The performance of the application is good enough with just a 1Gig Network (anything less and I agree performance is apparently so unimportant that we probably don't need to discuss it at all!) and 2) that if they were to measure it, the actual persistence framework code steals a fairly small bit of the overall application performance, making performance optimizations in the persistence framework a complete non-concern in their case. >Finally, think about the future. Network latency isn't a big problem. RAM isn't as well. But CPU speed is<
Or it could be that, given the requirements of an application, the CPU speed isn't a problem either. This is what I suggest is already the case for many apps. Yes, I realize that improving performance results in economic benefits - but the whole point here is that _other_ efforts (such as improving maintainability, etc) may yield even _more_ positive economic benefits after a point when sufficient performance has been acquired. (Cont)

Ayende Rahien commented on Benchmarks are useless, yes, again

Tue, 18 Aug 2009 12:09:15 GMT

Alex, All the numbers you throw around are _maybe_ accurate for lab situations. In practice, the numbers are tens of milliseconds.

Alex Yakunin commented on Benchmarks are useless, yes, again

Tue, 18 Aug 2009 11:48:26 GMT

> "at least 100K/sec rate per second" "at least 100K op/sec rate" :)

Alex Yakunin commented on Benchmarks are useless, yes, again

Tue, 18 Aug 2009 11:47:02 GMT

> You do agree with my logic, no? Yes, I agree with this. Let's think about other expenses we have now: - Index operations. I wrote Oren any high-performance server is equipped well enough to reply most of queries w/o HDD seeks. Either the whole database or its working set is cached in RAM. So almost no delay here: such operations can be performed with at least 100K/sec rate per second. - Network latencies. If you're caring about performance, 10Gbit channels are right for you. Latencies start from 2.5 microseconds there. Don't paste any links, since they're easy to find. So again, not an issue. - Network stack, or protocol latency. ~ 50 microseconds is latency reachable for WCF, so I suspect specialized protocols, such as VIA for SQL Server, provide much better one. 10-20 microseconds is fully ok for us here. So what else is left? I don't know. It seems I mentioned all show stoppers. So querying an enterprise storage with 10K op/sec in a single thread is more than possible (likely, even 20-30K op/sec isn't a limit). But NH can utilize only 1/10 of it on queries. I'm not speaking about CUD: ~ 10K commands per second * batch size (25 for DO) = 250K updates per second. So here we're easily reaching the limits of our framework. The same is true for NH. So... Have I shown all I shown must be visible even on very large storages (e.g. 0.1...1 TB)? Think about the smaller ones. I admit, 90-95% of databases that are currently used are <100Gb. Finally, think about the future. Network latency isn't a big problem. RAM isn't as well. But CPU speed is - i.e. they grow up only because of parallelism. But AFAIK neither NH nor any other ORM currently utilizes this (we plan).

Mats Helander commented on Benchmarks are useless, yes, again

Tue, 18 Aug 2009 11:46:19 GMT

Hehe, again my comment was too long - well good thing I have my new blog then ;-) Oren, I know you are every bit as familiar as I am with the potential for spectacle surrounding "PetShop" benchmarks, but please hear me out...I think you may possibly agree with me that there's a kind of Pet Shop benchmark that could be very useful. Please let me know what you think: [matshelander.blogspot.com/.../...ti-benchmark.html](http://matshelander.blogspot.com/2009/08/pet-shop-anti-benchmark.html)

Mats Helander commented on Benchmarks are useless, yes, again

Tue, 18 Aug 2009 10:19:29 GMT

>Ok, I see you're intentionally trying to hide pure NH performance by other numbers (I agree they might appear in high-stress conditions, but not everywhere), constantly saying "this is real, not what you show to us".<
That's not hiding it - that's putting the numbers into exactly the kind of context that you replied to me saying you agreed with was needed and you were working to fix. >I'll repeat myself: "So about importance of our tests: - CUD tests: important, if you deal with pretty limited sets of data. You'll still fill the difference on millions of records. But definitely not on billions. - Fetch, query: nearly the same. - Materialization: important in any case. If query is returning a part of the index, low-level RDBMS performance here is constant and very high (~ up to 20M of index entries per second by our tests - it's easy to measure this, just calculate an aggregate). So generally materialization is the main limiting factor in bulk reads."<
And I will also repeat myself: So what if the room for overall application performance improvment is, say, 5%? Then the points you mention above can convincingly demonstrate in tests that your framework is 100 times faster than mine, yet still you do ot manage to improve overall application performance by more than 1%. You do agree with my logic, no? /Mats

Alex Yakunin commented on Benchmarks are useless, yes, again

Tue, 18 Aug 2009 08:24:20 GMT

Oren, I see you don't want to answer just "Yes" or "No" ;) > You code would result in 1M+1 remote calls. We've already discussed this. I described this here: [ormbattle.net/.../...ur-tests-are-unrealistic.html](http://ormbattle.net/index.php/faqs/68-your-tests-are-unrealistic.html) > Put it on a reasonable network, and assume a 2 ms ping time, and your timing is going to so far off it isn't even funny. Ok, I see you're intentionally trying to hide pure NH performance by other numbers (I agree they might appear in high-stress conditions, but not everywhere), constantly saying "this is real, not what you show to us". I'll repeat myself: "So about importance of our tests: - CUD tests: important, if you deal with pretty limited sets of data. You'll still fill the difference on millions of records. But definitely not on billions. - Fetch, query: nearly the same. - Materialization: important in any case. If query is returning a part of the index, low-level RDBMS performance here is constant and very high (~ up to 20M of index entries per second by our tests - it's easy to measure this, just calculate an aggregate). So generally materialization is the main limiting factor in bulk reads." Do you agree at least with these statements? > Persistence by reachability isn't costly at all. It requires the same ~ O(N) scan time. So it's of the same complexity. > I think you are confusing dirty checking with that True - I imagined just graph traversal & forgot about this. So this must be much more computationally complex, but with the same ~ O(N) dependency.

Ayende Rahien commented on Benchmarks are useless, yes, again

Tue, 18 Aug 2009 08:16:02 GMT

Alexis, That sounds like a more reasonable approach for this, but see my comments about the PetShop benchmark stink when 1.0 came out

Ayende Rahien commented on Benchmarks are useless, yes, again

Tue, 18 Aug 2009 08:02:38 GMT

Alex, Persistence by reachability isn't costly at all. I think you are confusing dirty checking with that. As for implementing a different dirty checking method, the hooks are there, and it is quite easy to do.

Ayende Rahien commented on Benchmarks are useless, yes, again

Tue, 18 Aug 2009 08:01:16 GMT

Hendry, If that is the case, then the test should be using _different sessions_, not a single session

Ayende Rahien commented on Benchmarks are useless, yes, again

Tue, 18 Aug 2009 08:00:48 GMT

Alex, I suggest reading about the fallacies of distributed computing. You code would result in 1M+1 remote calls. Put it on a reasonable network, and assume a 2 ms ping time, and your timing is going to so far off it isn't even funny.

Ayende Rahien commented on Benchmarks are useless, yes, again

Tue, 18 Aug 2009 07:59:25 GMT

Alex, Your "test" is flawed once again, remove the idea, and try again. Try something that _resembles_ real world practices vs. a specially crafted thingie to show how it behaves.

Alexis Kochetov commented on Benchmarks are useless, yes, again

Tue, 18 Aug 2009 07:52:31 GMT

Hendry, Oren, we are going to implement TPC-C or TPC-E performance tests where all framework features will be allowed because it's business logic bench with huge DB workload. First stage is to make infrastructure for these tests (different implementations for particular ORMs should be rather similar). What do you think about it? Please reply.

Hendry Luk commented on Benchmarks are useless, yes, again

Tue, 18 Aug 2009 07:22:31 GMT

On the other hand, in all fairness, I don't think the point of the test is to intentionally make it OLAP. It merely magnifies OLTP operations in magnitute of hudrends-of-thousands times to observe how NH performs in high load, compared to other ORM products. Maybe the approach is not designed quite properly for NH, but probably someone could advice how to do it (high-load OLTP test) to provide more accurate result for NH.