Ayende @ Rahien

It's a girl

How to lose respect and annoy people

This post is going to be short, because I don’t think that at my current state of mind, I should be posting anything that contain any opinions. This is a reply to a blog post from X-tensive titled: Why I don't believe Oren Eini, which is a reply to my posts about the ORMBattle.NET nonsense benchmarks.

I will just say that if I was willing to let X-tensive enjoy the benefit of the doubt the reason they run the benchmarks the way they did, the post has made it all too clear that doubt shouldn’t be entertained.

Here are a few quotes from the post that I will reply too.

I confirm this is achieved mainly because of automatic batching of CUD sequences. It allows us to outperform frameworks without this feature by 2-3 times on CUD tests. So I expect I will never see this feature implemented in NHibernate.

That is interesting, NHibernate has batching support since 2006. The tests that X-tensive did never even bother to configure NHibernate to use batching. For that matter, they didn’t even bother to do the most cursory check. Googling NHibernate Batching gives you plenty of information about this feature.

Currently (i.e. after all the fixes) NHibernate is 8-10 times slower than EF, Lightspeed and our own product on this test.

Yes, for a test that was explicitly contrived to show that.

Since we track performance on our tests here, we'll immediately see, if any of these conditions will be violated. And if this ever happen, I expect Oren must at least publicly apologize for his exceptional rhino obstinacy ;)

Um, no, I don’t think so. Moreover, not that I think that it would do much good, but I still think that personal attacks are NOT cool. And no, a smiley isn’t going to help, not is something like this:

P.S. Guys, nothing personal.

Comments

Scott White
08/21/2009 04:48 PM by
Scott White

this company has proved they are classless and I will never buy their products.

MMoroz
08/21/2009 05:07 PM by
MMoroz

OMFG - this is i think the best way to summarize this flame.

Comparision of ORM-s based purely on performance? I understand that next week we are going back to discussion StoreProc vs ORM ;) - cause in terms of pure preformace SP pwn (^_^) every ORM.

So, if you think performance is the most important fact in choosing orm, than check your facts again.

Linq important? Yeah sure, but multiple database support, nested transactions, integrating with ADO.net ,easy integration with WF( or something else ), many others - these are really important. I can survive without linq, you miss the big ones and i won`t use your uber-fast/linqu-compiliant framework.

Van
08/21/2009 05:30 PM by
Van

For me, the most important thing is how the ORM affects my POCOs.

I'm now having taking over a project that most of the business logic was in stored procedure. I must say 90% of them.

They chose to use SP because:

  • High performance.

  • Reduced network traffic.

  • Transactional safe.

But by the time I take over the project, it was a big mess that nobody knows how it works.

The most simple update cause triggers, which in turn cause other triggers to run and a whole trunk of data is inserted into history take.

I don't believe in performance if I have to give up simplicity and integrity of the project.

Keith Baker
08/21/2009 05:52 PM by
Keith Baker

I couldn't agree with you more. Forget them and their trash. Keep up the great work.

Vadi
08/21/2009 06:51 PM by
Vadi

Oren, say the four word .. and forget them, they are crap !!

Richard
08/21/2009 07:10 PM by
Richard

How convenient that you cannot post comments on his blog.

I don't care how many smiley faces he adds, his tactics are, at best, slimy. He simultaneously [and blatantly] attacks his competitors while also using their branding and audience to further his own biased marketing site. Then he has the gall to retract some statements and change some code, all the while calling them innocent mistakes, while continuing to argue vehemently on others' blogs -- over, and over, and over, and over, and over, and over, and, well, you get the point -- that he is right, that he did nothing wrong and that he just trying to better the community.

Sorry, I am not buying it.

Then there's this little gem from his forums...

"Yes, Oren won the first match - we decided to remove DO from this web site. But the next one was lost: NH is still one of the worst products there. After all the optimizations they suggested (no more proposals just because they're stuck now - their test looks perfectly). So yes, we've lost one match, but the battle isn't finished."

It's sad that he doesn't even see what his need to win the war -- you know, the one that exists in his mind -- is doing his and his company's image.

ok
08/21/2009 07:31 PM by
ok

Nerd fight! Watch out, someone might break a nail!

Alex Yakunin
08/21/2009 07:55 PM by
Alex Yakunin

Oren, not sure if you seen this, but we turned on this feature almost immediately after our last conversation:

- code.ormbattle.net/
" <property25

"

  1. protected override void InsertTest(int count)

  2. {

  3. using (var transaction = session.BeginTransaction()) {

  4. for (int i = 0; i < count; i++) {

  5. var s = new Simplest(i, i);

  6. session.Save(s);

  7. if (i % 25 == 0) {

  8. // 25, same as the ADO batch size.

  9. // Flushes the batch of inserts and releases memory.

  10. session.Flush();

  11. session.Clear();

  12. }

  13. }

  14. transaction.Commit();

  15. }

  16. instanceCount = count;

  17. }

The info about this option is published e.g. here:

nhforge.org/.../...rovided-ado-net-connection.aspx

I'm not sure if this test is optimal for NH (moreover, current code is a bit different, i took this from Monday's snapshot @ code.ormbattle.net).

So we were aware about this option. We didn't closely study what really happens there, but it helped us to increase NH results ~ by 2 times. If there are any other issues, just notify me.

Alex Yakunin
08/21/2009 07:56 PM by
Alex Yakunin

Err... The blog just ate a quotation from .xml, so see the link.

Alex Yakunin
08/21/2009 08:00 PM by
Alex Yakunin

So the most current code:

protected override void InsertMultipleTest(int count)

{

  using (var transaction = session.BeginTransaction()) {

    for (int i = 0; i < count; i++) {

      var s = new Simplest(i, i);

      session.Save(s);

      if (i % 25 == 0) { 

        // 25, same as the ADO batch size.

        // Flushes the batch of inserts and releases memory.

        session.Flush();

        session.Clear();

      }

    }

    session.Flush();

    session.Clear();

    transaction.Commit();

  }

  instanceCount = count;

}
Alex Yakunin
08/21/2009 08:00 PM by
Alex Yakunin

adonet.batch_size = 25.

Alex Yakunin
08/21/2009 08:01 PM by
Alex Yakunin

Tomorrow I'll check what really happens on this test in profiler.

Rob Conery
08/21/2009 08:30 PM by
Rob Conery

OREN! Dude you're giving these clowns EXACTLY what they want - LINKS.

Alex - go back to your hole.

Harry S.
08/21/2009 09:22 PM by
Harry S.

+1 Rob, "All press is good press" as the saying goes. Maybe if we ignore them, they'll go away...

Richard
08/21/2009 09:26 PM by
Richard

Hi,

Ok, when I first saw the title of this blog post, I was thinking you were talking about yourself, because this has become my impression and I am not alone.

Your blog has long been interesting but in the past year it started to be more and more annoying and you lose the respect of quite a few.

Why ?

  • recurrent attacks against people. And critics, critics, critics...

  • never humble. You are NEVER wrong.

  • stuck sometimes in outdated programming paradigms (NHiberate being one of them seriously. There is far better tools for the job nowadays).

  • over-engineered pieces of code.

  • serious lack of openess and plain arrogance.

Still, I keep receiving you feeds, hoping for something interesting, but mostly reading entertaining flame wars you apparetly love.

But seriously you would elevate yourself by doing some serious introspection and think out of your box.

I wouldn't not be saying this if there was not much love behind. I care about people that matters to me.

Regards,

Richard

John Farrell
08/21/2009 10:34 PM by
John Farrell

With Oren and Alex being so passionate about their products did anyone think this would end in unicorns and rainbows?

The explosion of ORM minds, most notable, Alex, Frans and Oren left tons of great knowledge shrapnel in the comment threads which I've enjoyed reading.

The fact that Alex has gone back to his code and adjusted some things wins some points in my book. Some of you may think his speed comparison is unfair or slimey but you have to admin some 'non coding architect" is going to see his ORM's faster numbers and whip out his credit card. Slimey and Shrewd IMO.

Chris
08/21/2009 10:37 PM by
Chris

This is still going on? I cannot believe the crap that is being spouted.

I'd love to see them throw DO at my company's platform. I'd say we are probably on the larger end of NH users...

~400 domain objects ALL mapped via NHibernate.

peak 5000 concurrent users.

Many tables with tens of millions of rows

15 member dev team with 800+ SVN commits a month.

Massive multi-site redundant deployment.

Sub 200ms response times on all requests.

WHEN someone demonstrates performance and maintainability on that scale, rather than some silly argument over CRUD batching on a biased marketing site. There is no realistic data volume or concurrency on those benchmarks, therefore no credibility.

In the UK, it is against the law (ASA) to produce marketing FUD like OrmBattle.net.

gunteman
08/21/2009 11:10 PM by
gunteman

I had a look at DO quite some time ago, and found it to be an OK product, with some interesting concepts. But this marketing campaign disguised as a serious evaluation is really doing the product and its creators a disservice. Bad, bad move!

The personal attacks part? Well, I tend to agree with Richard. You're quite the mud slinger yourself, so a bit of backlash, once in a while, should be expected.

That's not to say I don't enjoy and benefit from most of your posts, but the ratio is going the wrong way.

Jimmy James
08/22/2009 12:01 AM by
Jimmy James

Now one of you gurus stands up and brings a starter kit that can handle any of these use cases with NH lightning fast.

Discussion s over.

Bob
08/22/2009 01:21 AM by
Bob

Why dont some of you guys stop whining and just TRY to use Alex's product in your real world app and see what it does.

All this sissy whining about whether it does or doesnt, and benchmarks or real world, whatever.

Shut your mouth, load his code, and prove it to yourself that it does or doesnt do something.

Then come back here and post about it.

jdn
08/22/2009 01:29 AM by
jdn

You seriously think this is a significant personal attack? Please.

In my book, in order for something to be a personal attack, it has to be insulting, and it has to be false.

For instance, suppose someone said:

"I think that jdn is kind of a dick."

Obviously, this could be taken as insulting, but it is also probably true. So, it isn't really a personal attack.

If someone calls you obstinate, the non-pejorative part of that is obviously true. You hold fast to your opinions.

I suppose the pejoritive part can be considered insulting and false. But only by a stretch.

Steve
08/22/2009 04:21 AM by
Steve

Bob, I think the point is to not post test data on performance of a product with vague numbers' prior to even knowing the optimal ways to run that product. ie. NHibernate has a Session-less state approach that will make batch peformance better. That wasn't used. Ayende posted that clearly. At the very least the test data should be re-run with the opitimal settings applied. (I'm sure the person who posted this survey used the best settings for his product to show how much better it is....)

What makes it questionable is when there is an obvious attempt here to skew the results because the person making these claims also has ORM product he is trying to highlight as being better.

That doesn't necessarily bother me, but to post data that is misleading is not...imo... not ethical and it's good that Ayende is calling them out on it.

Alex Yakunin
08/22/2009 06:12 AM by
Alex Yakunin

Ok, some results:

  • As I've mentioned, ADO.NET batching was turned on for NH in our tests from Monday. The results you see already "include" it.

  • Funny that I forgot to mention this in my post. I was aware about "adonet.batch_size = 25" option, but since it didn't help NH much (in comparison to DO, of course), my brain has lost this fact at all.

  • As you suspect now, batching in DO is quite different. I'll explain this in DO blog ( http://blog.dataobjects.net ) today, as well as publish a post in ORMBattle.NET blog describing the issue.

What we just did:

  • Eliminated stupid "if ((i%25)==0)" - it actually didn't change anything.

  • Checked the sequence in SQL profiler. There are 25 "RPC completed" on each 25th operation. That's exactly what ADO.NET batches imply.

  • Switched back to stateless session on InsertMultipleTest to see if there is any difference again. We tried this earlier here, but didn't seen a noticeable one (earlier on this week we switched back to regular session: I wrote I consider stateless session usage as special API, so we're avoiding this).

Let's see what we have right now on 1K sequence:

DataObjects.Net

Create Multiple: 20105 op/s

Update Multiple: 35018 op/s

Remove Multiple: 42542 op/s

NHibernate

Create Multiple: 12617 op/s

Update Multiple: 17148 op/s

Remove Multiple: 18510 op/s

As you see, IStatelessSession brings ~ additional 10% to shown results here.

Test code:

protected override void InsertMultipleTest(int count)

{

  using (var statelessSession = factory.OpenStatelessSession())

  using (var transaction = statelessSession.BeginTransaction()) {

    for (int i = 0; i < count; i++) {

      var s = new Simplest(i, i);

      statelessSession.Insert(s);

    }

    transaction.Commit();

  }

  instanceCount = count;

}


protected override void UpdateMultipleTest()

{

  using (var statelessSession = factory.OpenStatelessSession())

  using (var transaction = statelessSession.BeginTransaction()) {

    var query = statelessSession.CreateQuery("from Simplest").List

<simplest();

    foreach (var o in query) {

      o.Value++;

      statelessSession.Update(o);

    }

    transaction.Commit();

  }

}


protected override void DeleteMultipleTest()

{

  using (var statelessSession = factory.OpenStatelessSession())

  using (var transaction = statelessSession.BeginTransaction()) {

    var query = statelessSession.CreateQuery("from Simplest").List

<simplest();

    foreach (var o in query)

      statelessSession.Delete(o);

    transaction.Commit();

  }

}

And about "P.S. Guys, nothing personal." - Oren, I think you should quote a bit more in this case. There is:

"P.S. Guys, nothing personal. I love reading your blogs and I know you're quite clever persons. But I think if you're standing on your point of view so hard, you are ready to prove it in action. Quite simple."

Nice that you didn't answer the main question. I understand attracting the attention to arguable details is the best tactics here. So this won't work.

Alex Yakunin
08/22/2009 06:23 AM by
Alex Yakunin

And about personal attacks: I say once more there is nothing personal. We've been testing NH. You're its developer. You're protecting it hard, and get similar responses.

That's all around argue itself. "Attack" = "false statement about you personally". At least for me. And the title of your post, btw, partially fits to this.

  • "Why I don't believe Oren Eine" - here I explicitly mention this is solely my opinion.

  • "How to lose respect and annoy people" - I see nothing like this here. You tell this like a mentor, or expose an aggregated option.

Alex Yakunin
08/22/2009 06:44 AM by
Alex Yakunin

So I feel you're intentionally try to re-classify it as "personal attack". It's the easiest way to show I'm bad guy, and you're the good one.

That makes it easier to make the people believe the results I show you are useless. People need reasons to believe in something, that's a part of our psychology. "Alex personally attacks me (= he is bad)" is the one that will be accepted here with ease - I clearly understand I'm at NH territory here.

So I suggest you to discuss just tests and results further, and forget about anything else for some period. It will be much easier.

Alex Yakunin
08/22/2009 06:48 AM by
Alex Yakunin

And, Oren, I clearly understand this Nth case for you when someone else tried to compare his own product with NH. So I partially understand why you don't even look into the tests. You believe IT'S ALL THE SAME CASE.

But it's different, and not just this one, but each time you get such a case. Although your mind tend to think you can apply some "statistics" you have here. You compete with all the world now. And you must accept the fact that someone might win you one day.

Alex Yakunin
08/22/2009 06:51 AM by
Alex Yakunin

Latest test results are already committed to repository (/Output.txt).

Alex Yakunin
08/22/2009 07:13 AM by
Alex Yakunin

@ Richard: thanks. Btw, imagine, how I feel myself talking with Oren. I'm curious, was the same happening with others.

Mark
08/22/2009 01:49 PM by
Mark

Regardless of the perceived personal slights, I have fundamental misgivings about the approach of ORMBattle.NET.

Yes, you can find out that the site is run by folk who write an ORM product, but to anyone who comes to the site, it initially gives the impression that it is an independent comparison site. Regardless of intent, you can never be 100% independent. Would you trust a paper funded by a company that has a vested interest in the results? I certainly would not.

I'm not saying that the results are definitely skewed or intentionally biased, but I can't trust them whatsoever due to the conflict of interest. It looks bad.

Alex Yakunin
08/22/2009 04:44 PM by
Alex Yakunin

@ Mark: Check out the rules: http://ormbattle.net/index.php/rules.html

I'm thinking how to make this much more independent. Voting based on participant voices is one of possible ways. If you know a better one, just list it.

Yes, we launched this site, and this can't be neither hidden nor ignored. But it is in our forces to make it much better. We'll discuss any proposals.

As I've said, I believe such a resource must exist.

Jonatan
08/22/2009 06:02 PM by
Jonatan

Oren: There is no "winning" this. Leave it be. They can claim that they are so much better then NHibernate. Let them. I think you have said what was needed to say. The tests have nothing to do with real world performance and they are not relevant when comparing ORMs. The tests are useless. Move on and don't give them any more publicity.

Alex Yakunin
08/22/2009 06:07 PM by
Alex Yakunin

Nice point ;) But I think Oren must accept proposed bet first. If so, I won't comment this any more here.

Mike Griffin
08/22/2009 06:19 PM by
Mike Griffin

Alex, I posted on that thread and gave your suggestions for the single tests, I think those would be very useful and consistant across all ORM solutions. I think if you like those tests we can probably participate, however, we still object to having transactions being used when doing mulitple fetches and reads (and thereby using the same connection allowing for special optimizations to take place, folks don't normally wrap their queries in transaction, and then to do 100 queries within the same transaction, not very useful or real world. We're making progress though, look at my single tests suggestions at the bottom of this post on your forums

ormbattle.net/.../...like-to-in-orm-battlenet.html

I believe the tests can be useful, adopt those single tests, dump the transactions around reads, and it will be hard for folks to complaign ...

Jonatan
08/22/2009 06:31 PM by
Jonatan

Alex: No he doesn't have to accept your bet. You may want him to but I don't think he has any kind of obligation to do it. There is no point in continuing this. Everything Oren responds to in any way you have used as a way to continue this debate. I propose that he just stops responding to your'e comments.

Jeff
08/22/2009 10:05 PM by
Jeff

This Alex guy has proved himself a marketing idiot for his product. Even if it was faster, better whatever, I would use another vendor because of his attitude on your blog and others. There are plenty of alternatives...

Ayende Rahien
08/23/2009 09:33 AM by
Ayende Rahien

Alex,

Doing batching at the TDS level is far more efficient. It also protect you from issues with max number of parameters per call.

There are other disadvantages to doing things this way as well, you create higher load of the query plan cache on the DB server and incur higher parsing costs.

NHibernate had support for this for reads for almost 4 years now. It is called Multi Query and we extended the support in 2.1 for full Futures implementation.

This is a conscious decision that the user must take, because of the issues with the query plan cache that I mentioned.

For writes, batching at the lower level that we do is more efficient.

Although, if you really wanted, it would be pretty easy to write another Batcher to support this.

cowgaR
08/23/2009 11:16 AM by
cowgaR

the site code.ormbattle.net is marked as DANGEROUS by Firefox 3.x & Google (red warning Firefox page)...

google is saying it is installing trojans... get this right :D

@Alex: do you support POCO? I have no problem using your framework if you support it, or at least can show me an easy way to test my entities (I understand there is some dependency going on with all those attributes). Or is there apple & oranges comparison going on?

Mike Griffin
08/23/2009 12:48 PM by
Mike Griffin

There are some bizarre comments on this thread. Like stored proc's "OWN " ORM's? How bizarre, we support stored proc's or dynamic SQL, you indicate which you want via the connection, no code changes what so ever. Stored procs and ORM's are not mutually exclusive. Also, battle ORM is not marked as DANGEROUS by google or firefox 3.x, just checked, anyway, have a good day everyone ...

Benetton
08/24/2009 03:45 AM by
Benetton

IMHO:

It's just so ridiculous to read some guys posting stupid things like "I'd never buy this Alex's ORM". What's the point?

It's not about buy/not buy - it's about speed, speed and speed.

Everyone who's crying about anything else (scalability, poco, unit of work, bla bla bla) is cheating here.

Oren - if your gorgeous NHhibernate is a hammer (as your picture assumes), then I would never use it. Softdev is no house-building - and I don't need any stupid hammers in my toolset.

P.S. Looks like the only cause of all these massive insults here is Oren's inflamed ego.

Dave Mertens
08/24/2009 07:04 AM by
Dave Mertens

The tests at ormbattle are wrong because they benchmark something you will never use in that way. Deleting 10.000 rows? Why on earth would I first retrieve 10.000 records from the database only to delete them as a next step? Executing a 'delete from TableX' directly on the database will outperform every ORM.

Our applications often have to work in legacy environments. So I need a ORM framework that doesn't require changes to the legacy code to use them. Most ORM framework require you to either inherit from a base class (like ActiveRecord) or to mark your class and properties with attributes (like Linq to SQL). The only 'downside' of NHibernate in this approach has is to every class (in the hbm.xml) have to marked 'lazy="false"' to methods and properties doesn't have to be virtual. I also have used Linq To SQL with XML mapping files (XmlMappingSource) on the same projects. Both are still in use and serve different needs.

Alex Yakunin
08/24/2009 01:29 PM by
Alex Yakunin

@ Oren: about "Multi query + CUD batching" vs "generalized batching" - I hope we cleared everything up by e-mail. I can copy my answers here, but don't want to spend the time on this any more.

Let readers make their own decisions & conclusions.

Ayende Rahien
08/24/2009 01:33 PM by
Ayende Rahien

@Alex,

Removed comment that contained only links

Alex Yakunin
08/24/2009 01:58 PM by
Alex Yakunin

Nice tactics, Oren. Ok, let's say this another way:

@ Dave: see "FAQs" section @ ORMBattle.NET:

  • "Why you run a single operation for multiple times during the measurement?"

  • "In fact, you're testing bulk insert \ update \ remove operations in "CUD Multiple" tests. Why you prevent usage of executable DML queries there?"

  • "Is it possible to use a specialized API instead of common in tests?"

Alex Yakunin
08/24/2009 02:00 PM by
Alex Yakunin

It's not about buy/not buy - it's about speed, speed and speed.

Everyone who's crying about anything else (scalability, poco, unit of work, bla bla bla) is cheating here.

At least, it seems very close to true.

Peter
08/27/2009 04:05 AM by
Peter

@Bob: Exactly!! EntitySpace's attitude to this whole thing is the right one imo.

Bunter
08/27/2009 01:01 PM by
Bunter

Everyone who's crying about anything else (scalability, poco, unit of work, bla bla bla) is cheating here.

At least, it seems very close to true.

Yeah, right. Only l00t lamrz care about those things. Name the site ORMBattle.net and do only performance tests. The size of the userbase, open source-ness, flexibility, testability, third party support... I mean who cares about those things.

Chris
08/30/2009 07:17 PM by
Chris

Hey at least DO and NH are not Entity Framework :)

Alex Yakunin
08/31/2009 04:29 PM by
Alex Yakunin

I mean who cares about those things.

That's simply a different question. We really don't answer it. On the other hand, this kind of information is much easier to find.

Oren, that's for you: technet.microsoft.com/en-us/library/cc966425.aspx

Quote: "Comparison of Recompilations in SQL Server 2000 and SQL Server 2005

When a batch is recompiled in SQL Server 2000, all of the statements in the batch are recompiled, not just the one that triggered the recompilation. SQL Server 2005 improves upon this behavior by compiling only the statement that caused the recompilation, not the entire batch. This "statement-level recompilation" feature will improve SQL Server 2005's recompilation behavior when compared to that of SQL Server 2000. In particular, SQL Server 2005 spends less CPU time and memory during batch recompilations, and obtains fewer compile locks.

One benefit of statement-level recompilations should be obvious: it is no longer necessary to break a long stored procedure into multiple short stored procedures just to reduce recompilation penalty of the long stored procedure."

Ayende Rahien
09/01/2009 09:04 AM by
Ayende Rahien

Alex,

NHibernate works on more DBs than just SQL 2005, you are aware of that, right?

Alex Yakunin
09/01/2009 09:20 AM by
Alex Yakunin

Right. But e.g. PostgreSQL behaves nearly the same. And we didn't run similar performance tests with Oracle yet.

Oren, I don't want to push you to do anything - that's just FYI.

Btw, we've published an updated test suite & its results today. LINQ tests results are now categorized + there is new paging test.

Comments have been closed on this topic.