Ayende @ Rahien

Alex Yakunin commented on Benchmark cheating, a how to

Fri, 21 Aug 2009 05:36:24 GMT

Oren, I'd like to notify you I've made a post into ORMBattle.NET blog related to our discussions: [ormbattle.net/.../...i-dont-believe-oren-eini.html](http://ormbattle.net/index.php/blog/88-are-results-of-this-test-suite-really-important-part-1-why-i-dont-believe-oren-eini.html) P.S. There is a good FAQs section now. So if you disagree with something, please check it first. Although it's mainly based on our debates here (no any quotes, of course).

Alex Yakunin commented on Benchmark cheating, a how to

Thu, 20 Aug 2009 06:14:34 GMT

Sounds reasonable, especially for JIT.

Ayende Rahien commented on Benchmark cheating, a how to

Thu, 20 Aug 2009 05:26:24 GMT

Alex, No, it won't. If would optimize the case of arr[i], but not the case of arr[5]. The reason for that is that in a loop, you _very rarely_ do arr[constant or variable that isn't loop var] It doesn't make sense to try to optimize that. In fact, it is a bad practice, because you are spending effort on something that won't happen. You are adding complexity and not getting much in return

Alex Yakunin commented on Benchmark cheating, a how to

Thu, 20 Aug 2009 05:10:09 GMT

Looked it up more carefully, and now I'm not fully sure about this: I'm 100% sure it will eliminate all unnecessary bound checks for arr[i] in this case. But you wrote arr[const] = i; so on one hand, this is even a simpler case; on another, a lot depends on JIT logic here. Anyway, it was just a remark to your example. I understood what you wanted to say, of course.

Alex Yakunin commented on Benchmark cheating, a how to

Thu, 20 Aug 2009 05:05:45 GMT

Oren, I'm writing FAQ now, and found your example: "C++ int[] arr = new int[100]; for(int i=0;i<100;i++) arr[5] = i; C# int[] arr = new int[100]; for(int i=0;i<100;i++) arr[5] = i; And then pointing the perf difference related to array bound checking in a contrived scenario. There isn't a point in trying to fix anything, the scenario that you are using is broken." I'm curious, do you know that exactly in this case .NET will run just 1 array bound check (one per the whole loop)?

Ayende Rahien commented on Benchmark cheating, a how to

Tue, 18 Aug 2009 08:58:39 GMT

> I tried to wriggle out of it by referring to map reduce LOL

Mats Helander commented on Benchmark cheating, a how to

Tue, 18 Aug 2009 08:58:06 GMT

>> use cases keep growing hairier and databases keep growing new tables >You miss the point, it isn't # of entities that is important, it is # of entity _instances_ loaded into the session, and it is only noticable if you run something like this, which is sitting in a tight loop forcing this to happen over and over again. Sorry I was unclear here. I simply meant that as more tables find there way into a use case, chances are that you'll end up having more instances in it, too...

Mats Helander commented on Benchmark cheating, a how to

Tue, 18 Aug 2009 08:54:19 GMT

Thanks for the detailed reply, I don't fully agree with you about whether it is a good default behavior to use comparison based dirty tracking, but I do appreciate that it is a trade-off and that your arguments make complete sense - again, it is a case where I value things differently in a trade-off, I suppose neither of us is more "right" but I'm always happy to better understand the technical aspects of the trade-offs so thanks again for the detailed reply. Btw, regarding the flushes - Roger Alsing just pointed out to me that I get a problem with my approach in where clauses with aggregate constraints - true enough and so here I must concede that N/H does the right thing, after all! I tried to wriggle out of it by referring to map reduce but I don't think he bought it ;-)

Alex Yakunin commented on Benchmark cheating, a how to

Tue, 18 Aug 2009 08:28:01 GMT

Than let's forget about this. I fairly believe money will change something here. But if they would... This would depreciate the whole discussion. Anyway, thanks for advices.

Ayende Rahien commented on Benchmark cheating, a how to

Tue, 18 Aug 2009 08:13:57 GMT

Mats, > The reason NH is slow here is that it can't do dirty tracking by relying on notifications from the domain objects Yes. > About N/H doing that flush...it has always seemed strange to me: The idea here is that you'll get the in memory results from the database in you make a query. It is confusing for the user if this doesn't happen. > using a compile time weaver (or factories for runtime subclassing) Yes, and if you want to do that, NH contains the hooks to provide it. > use cases keep growing hairier and databases keep growing new tables You miss the point, it isn't # of entities that is important, it is # of entity _instances_ loaded into the session, and it is only noticable if you run something like this, which is sitting in a tight loop forcing this to happen over and over again. > how do I get my domain objects to issue the notifications? However you want to do so. A common approach is to do it using IsDirty flag that is weaved in. NH requires that you will override Interceptor.IsDirtyFlush() and provide an implementation, it doesn't dictate how. > And I have to say I do wonder why it is not the default approach. Because it places constraints on the domain. And it is fragile. For example, imagine a method customer.DoSomething() that updates a field directly. That DoSomething() method needs to set the IsDirty flag. It is either that (which is very fragile), use AOP and force only property setters or try to intercept every field set. Any of those have significant issues associated with them, so they aren't the default. > Why not just start by taking the results from the DB Because you might not get the values that you would get if you flushed first. Imagine that the name is not set to the value you are selecting in the DB but is set to it in memory. By flushing it, you are ensuring that it is returned from the query. It is also not possible to run this queries in memory, imagine that there is a trigger in the DB that change the value, or aggregation that requires DB values, or a SQL function, or... you get the point. > This would have the advantage of working outside a transaction: That is not an advantage, NH goes to a lot of trouble to be transactional. And if you don't want this feature, you set the FlushMode appropriately.

Ayende Rahien commented on Benchmark cheating, a how to

Tue, 18 Aug 2009 08:03:48 GMT

Alex, No, I am not going to confirm near optimal code. I am going to follow Fabio's suggestion, if you would like my NHibernate expertise, please contact me privately to discuss my rates.

Alex Yakunin commented on Benchmark cheating, a how to

Tue, 18 Aug 2009 08:01:00 GMT

@ Mats: Short answer: [ayende.com/.../...marks-are-useless-yes-again.aspx](http://ayende.com/Blog/archive/2009/08/15/benchmarks-are-useless-yes-again.aspx#33387)

Alex Yakunin commented on Benchmark cheating, a how to

Tue, 18 Aug 2009 07:56:52 GMT

> think the part that is missing from your benchmark to give a useful, big picture understanding of what the results imply, is a clear explanation that put the numbers into the context of the overall optimization potential. Fully agree. I'm working on this...

Mats Helander commented on Benchmark cheating, a how to

Tue, 18 Aug 2009 07:49:29 GMT

@Alex, I think the part that is missing from your benchmark to give a useful, big picture understanding of what the results imply, is a clear explanation that put the numbers into the context of the overall optimization potential. As others have pointed out, in a real project where you start involving databases on dedicated servers and business logic code on the application server that does more than run a loop, then there is the very real potential for any optimizations on the O/RM code to become relatively cost inefficient. What I mean is: Say that you write a "real" test application (say, a PetShop or whatever) and you measure that the _worst_ of the frameworks that you test only gives, say (pulling numbers out of my a** here for sure, but bear with me) 5% overhead when compared to a corresponding application using only barebones ADO - the reason being that even if lots could be optimized in the framework, the time it consumes is dwarfed by the time it takes to run over the network to the database, etc etc) In such a scenario, there would just not be much opportunity for a framework to try to compete on raw speed, since the best it could hope to do would be to shave a bit off that 5% overhead. Even a magical framework that cut it down to 0% wouldn't make much difference to the overall performance (only 5%, as it were). In such a scenario, customers would probably be better off searching for the framework that provided good documentation, feature richness, ease of use and so on, since trading all that away for a measly 5% speed improvement just wouldn't make much sense. Your framework could be 100 times faster than mine and it might still just translate to an overall 1% overall performance improvement in your application. Now, you may protest that the span to the worst (interesting) competitor is larger than 5%, making the speed issue more important. But without seeing any tests to that effect, how do we know? That's why I think for a complete and honest picture, you should try to complement your tests with a discussion around, and a testing of, how big this type of overhead really is in a typical application.

Alex Yakunin commented on Benchmark cheating, a how to

Tue, 18 Aug 2009 07:32:08 GMT

Btw, can you confirm there is a near-optimal test code for NH now? Or there is something else left? Here it is: [code.ormbattle.net/](http://code.ormbattle.net/#&&target=Tests%2fPerformance%2fNHibernateTest.cs)

Mats Helander commented on Benchmark cheating, a how to

Tue, 18 Aug 2009 07:30:46 GMT

"And the reason that it is O(N) operation is because NHibernate doesn't place constraints on persistent classes." And these (lack of) constraints are that N/H doesn't rely on a factory nor a compile time weaver? So I was correct to assume that *is* the "feature" responsible for the non-linear times, then? ;-) To recap: The reason NH is slow here is that it can't do dirty tracking by relying on notifications from the domain objects (because it doesn't want to rely on factories or compile time weavers?). Instead you have to do dirty tracking by comparing the states of all the loaded objects to their original values. This of course gives the consequence that it becomes slow in scenarios where a lot of objects are loaded and you have to check them all. That N/H wants to check if it should flush to the DB in this very scenario is _another_ design decision of N/H that has continued to puzzle me (*) but it is not really the issue here - the issue is that the decision to default to "comparison based" dirty tracking becomes slow whenever you have to check a lot of loaded objects, and this is just one example where this issue becomes a concern, since that check is triggered. Again, using a compile time weaver (or factories for runtime subclassing) so that you'd get notification based dirty tracking would completely solve this, right? (Yes, N/H might still want to flush to handle the example in your post, but then this would be fast and linear) "and something that only show up as a problem when you start dealing with _large_ number of entities." Of course. But that does happen eventually when use cases keep growing hairier and databases keep growing new tables ;-) But you are completely right of course that a lot of the time, this just isn't an issue! "the second is to override NHibernate's dirty checking using the interceptor." By using a compile time weaver on the side? Or how do I get my domain objects to issue the notifications? Now, of course supporting the scenario where for some reason you can't use factories or weavers is nothing to be sneezed at - on the contrary, I think it is *quite* cool that there's a framework that does that (it certainly speaks to the purist in me!) but if NH really supports a notification based solution "out of the box" then I guess that's definitely something that should be taken into account by Alex's coders, because I imagine it will change numbers dramatically (and most importantly, make them linear). And I have to say I do wonder why it is not the default approach. (*) About N/H doing that flush...it has always seemed strange to me: Why not just start by taking the results from the DB and then checking the dirty objects in memory to see if any modifications should be made to the results? This would have the advantage of working outside a transaction: in a transaction you can roll back that flush, but if no transaction is active, what says I really want those dirty objects in memory going back to my database - it could well be that I want the fresh data from the DB to be able to determine that it would be a *disaster* for me to commit my UoW, right? :-) I do know that actually implementing this strategy used to be a hassle (since it basically required you to write a provider running your queries against an object model), but nowadays, with everyone running LINQ, LINQ to Objects does pretty much this for free, right? (I'm sorry if this is something you have already answered a 100 times, or even once, if so please link, I haven't been able to see anything on the web properly addressing this question) /Mats

Alex Yakunin commented on Benchmark cheating, a how to

Tue, 18 Aug 2009 07:27:44 GMT

> NHibernate doesn't place constraints on persistent classes And if there are no constraints, it uses persistence by reachability pattern. Quite costly, yes? E.g. I'd prefer to forget about "persistence for every built-in type" (this is anyway not true) and deal with tracking or base types, rather than deal with such issues (just disabling flushes before queries isn't a solution - it's a workaround). > EF, for example, requires a base class and managed fields, so it can notify the context about dirty checks. You know, it's possible to implement this without a base class. But you must communicate with tracking service in this case.

Ayende Rahien commented on Benchmark cheating, a how to

Tue, 18 Aug 2009 06:54:28 GMT

Mats, The reason for it being slow is that NH needs to check if a flush is necessary. And the reason that it is O(N) operation is because NHibernate doesn't place constraints on persistent classes. As such, dirty checking must be done by NHibernate, and that requires scanning all persistent entities and comparing all their fields. EF, for example, requires a base class and managed fields, so it can notify the context about dirty checks. NH needs to perform the check explicitly. This is a trade off for making the model freer from constraints, and something that only show up as a problem when you start dealing with _large_ number of entities. There are several ways of working around that, one is the flush mode setting, the second is to override NHibernate's dirty checking using the interceptor.

Mats Helander commented on Benchmark cheating, a how to

Tue, 18 Aug 2009 06:29:56 GMT

Hi Oren, I have to confess that again, I don't think I get it: You say that 1) the reason NH is slow is that it has to check if it needs to flush and 2) the reason this flush is slow is that it needs to go check all the objects in the identity map and that 3) The reason all this is good is because it allows NH to handle the example code in your post correctly. But (as Alex has pointed out) other frameworks also handle your code example correctly, but without the non-linear overhead. Thus, it can't be *that* feature (being able to handle the code example you show correctly) which is in itself responsible for the non-linear overhead - rather it must be either: 1) NH has made some suboptimal architectural decisions (and hey, who doesn't make those, I'm won't try to sound innocent!) or 2) There is some *other* feature that NH _can_ do but that the other frameworks with linear response times _can't_ . If this is the case then I am really curious what that feature is, because actually this very design decision of N/Hibernate has puzzled me since as long as I can remember. Unless I am mistaken, the "feature" that it actually does enable is that it allows you to use "new()" to create domain objects, so it does _not_ require using factories (for runtime proxying of all domain objects) and _also_ does not use a compilation time AOP solution that injects the necessary code for dirty tracking right into the original domain objects...though tbh whether not using something like that can be thought of as a feature is not clear to me). Please correct me if I am wrong? /Mats

Ayende Rahien commented on Benchmark cheating, a how to

Tue, 18 Aug 2009 01:31:42 GMT

FallenGamer, Look at Sharp Architecture as a good example.

Alex Yakunin commented on Benchmark cheating, a how to

Mon, 17 Aug 2009 21:19:59 GMT

Well, yes. Apply is supported only by SQL Server. On the other hand, this is quite powerful construction: e.g. its presence eliminates the need of joins. Frankly speaking, I don't see any reasons for not supporting it by other RDBMS: almost any of them supports subqueries (the underlying processing there must be almost the same as for APPLY), but we were unable to find APPLY analogue nether for Oracle, nor for ProsgreSQL. Anyway, its absence makes ORM vendor's life much more complex.

Frans Bouma commented on Benchmark cheating, a how to

Mon, 17 Aug 2009 20:14:28 GMT

Thanks for removing us. About the Apply stuff... only sqlserver has that operator, it's only needed if you do cross-join operand filtering, which is actually pretty rare, IMHO. I have to check what we do internally (I think we either give up or try to rewrite it to something which does work, if possible with a custom filter in the on clause).

Alex Yakunin commented on Benchmark cheating, a how to

Mon, 17 Aug 2009 20:07:06 GMT

Oren, if there will be any further suggestions on how to improve NH results, you're welcome. As well as any other people ;)

Alex Yakunin commented on Benchmark cheating, a how to

Mon, 17 Aug 2009 20:04:20 GMT

So: [http://ormbattle.net/index.php/blog.html](http://ormbattle.net/index.php/blog.html)

Alex Yakunin commented on Benchmark cheating, a how to

Mon, 17 Aug 2009 19:38:46 GMT

I don't know all the details (Alexis Kochetov is the guy responsible for this), but I'm 100% sure there are no IN queries: - List of items there is normally limited - This works only for simple keys. On the other hand, I don't know what exactly happens now. Let's leave discussion of how this works till the moment this is fully done. As I said, what we have currently looks more than trick than like a real implementation. Concerning the complexity of implementation of all this stuff: I can mention our query transformation pipeline is likely one of the most advanced ones. An example of one of transformations we do is here: [blog.dataobjects.net/.../...n-pipeline-inside.html](http://blog.dataobjects.net/2009/07/query-transformation-pipeline-inside.html) (that's one more reason of test failures on other frameworks except EF & DO4) So I think we'll be able to implement EL. And I fully agree, this is a "must have" feature for any serious player here.

Frans Bouma commented on Benchmark cheating, a how to

Mon, 17 Aug 2009 18:29:45 GMT

But does your eager loading also tweak sub node queries on the fly? I.e.: to fetch subnode (by filter on parent), do you use simple IN clauses when possible and if not switch to subqueries? eager loading at first isn't that difficult, to get it working with different graph setups (e.g. lots of parents, single child, not a lot of parents, many children etc.) is another story. I really am surprised that you didn't implement this o/r mapper must have feature. Especially in distributed scenario's, eager loading is essential.

Alex Yakunin commented on Benchmark cheating, a how to

Mon, 17 Aug 2009 17:27:52 GMT

> Filtered, ordered eager loading graphs, in LINQ. Doing that by hand is really really expensive and slow. I fully agree with this. But: - There is no standard for this. I.e. it's your own extension of standard. How can we test this? - And implementation of prefetch and quality of LINQ standard implementation are two different things. We test the second one, since it's obvious how to test it, and it's meaningful. Do you know how to do the same with prefetch? So when they'll appear in DO4 (I think we need ~ month to get this fully working), obviously, we won't include this into any of such tests. Finally, although I said we don't support EL, we support one more nice feature: while materializing query result, we load & materialize all the entities which properties are exposed in the final projection there. This provides _some_ support for EL, and for now this is acceptable. I think Oren will approve NH's LINQ is even worse from this point, but this is also acceptable.

Alex Kofman commented on Benchmark cheating, a how to

Mon, 17 Aug 2009 17:18:05 GMT

Surely eager loading is essential feature for OR mappers, especially when we a talking about ORM performance. It will be interesting to compare your implementation with another. I personally would like to see some comparison or overview.

Frans Bouma commented on Benchmark cheating, a how to

Mon, 17 Aug 2009 16:49:58 GMT

"We don't support eager loading so far - i.e. you should do this fully manually. But this part is in development ;}" you don't ? I'm very surprised! Also not in nested queries in linq projections? And you have a big mouth about linq provider quality concerning my work (which you suggested would fail if it was handed non basic queries) ? :) If something is good for every day real life performance it's eager loading. Example: var q = (from c in metaData.Customer where !(new string[] { "FISSA", "PARIS" }.Contains(c.CustomerId)) select c).WithPath(customerPath => customerPath .Prefetch c.Orders).OrderByDescending(o => o.OrderDate).LimitTo(2) .SubPath(orderPath => orderPath .Prefetch(o => o.OrderDetails) .Prefetch(o => o.Employee))); filtered, ordered eager loading graphs, in linq. Doing that by hand is really really expensive and slow. In real life scenario's that is of course. Perhaps you should add 50 or so linq queries which fetch graphs, filter the sub graphs etc. That would skew the results a bit, eh? :) >

Alex Yakunin commented on Benchmark cheating, a how to

Mon, 17 Aug 2009 16:47:36 GMT

> If you want this or/m battle marketing campaign to take off at least make the examples a bit more compelling. Seeing that your framework is good at something that I don't need doesn't really get me all that excited about it. I understood the following: - Some people clearly understand such tests as CUD & materialization. - Others need some real tests, and this site completely misses this part. So as I've mentioned, we already decided to add standard TPC tests there. Hopefully this will fix the issue.