Ayende @ Rahien

It's a girl

A question of an untenable situation

One of the most common issues that I run into in my work is getting all sort of questions which sound really strange. For example, I recently got a question that went something like this:

What is the impact of reflection on NHibernate’s performance?

I started to answer that there isn’t one that you would notice, even beyond the major optimizations that NH had in that role, you are accessing a remote database, which is much more expensive. But then they told me that they profiled the application and found some stuff there about that.

I asked them what their scenario was, and the exchange went like that:

Well, when we load a million rows…

And that is your problem…

To be fair, they actually had a reasonable reason to want to do that. I disagree with the solution that they had, but it was a reasonable approach to the problem at hand.

Comments

PS
07/29/2010 09:34 AM by
PS

That's why we have CQRS. Loading a million rows via NHibernate is insane (been there, done that, failed)

Howard van Rooijen
07/29/2010 09:42 AM by
Howard van Rooijen

I hate the fact that there is this huge problem with refelction in .NET - there was a problem in 1 / 1.1 - 7 odd years ago - that it wasn't as optimised as it should have been - but in following releases of the framework - reflection has been really tuned. People always seem to jump and point the finger at reflection - rather than profile and measure their own code / logic / process. I can guarentee it's gone through a lot less scrutiny and optimisation than the CLR / BCL.

tobi
07/29/2010 12:12 PM by
tobi

And I guess HN generated code to make typical reflection accesses faster like materialization or extracting all property values of an object into an array. The difference can only be measured in micro-benchmarks.

Andrew
07/29/2010 04:31 PM by
Andrew

Why should loading million rows be a problem? It's not that many; anyway, the need to deal with massive amount of data is not that unusual outside of the typical customer-orders pet stores. Fortunately, there are frameworks around (like BLToolkit) that give you flexibility to efficiently do whatever you want without saying "you're doing it wrong!" or "you won't notice the performance hit".

Ayende Rahien
07/29/2010 08:48 PM by
Ayende Rahien

Andrew,

million rows is generally an indication that you aren't doing business logic. You are generally doing reporting, ETL, etc.

That is something that requires different approaches than OLTP

Steve Py
07/29/2010 11:15 PM by
Steve Py

Or doing a mass-update that is much more efficiently handled via an update statement or stored proc. An ORM is a good hammer, but that doesn't mean you have to build an application without a screwdriver or a saw.

I've seen it plenty of times, loading huge #'s of objects just to set some relevant state and save them again.... Because it's consistent that way.

Daniel Fernandes
07/30/2010 07:20 AM by
Daniel Fernandes

Exactly the situation a project I'm joined.

NHibernate + 2nd level cache are used for a financial ETL + reporting engine where we're having to deal with million of rows generated daily and where the process takes over 6 hour to complete.

NHibernate is clearly more suited to OLTP scenarios with as small UoWs as possible.

And no, CQRS doesn't necessarily help for pure ETL scenarios.

Possibly RavenDb could be suited for this kind of scenario where it handles streams of data which gets processed over a period of time through map reduce functions.

Daniel

Daniel Kreß
07/30/2010 08:09 AM by
Daniel Kreß

Talking about a million rows...

Is it insane to use NHibernate to work on huge amounts of data like this...

Scenario:

  • I convert a DB table with exactly 1.000.000 rows by code, maybe with some joins :)

  • Using Linq to Entities with the Option "No change tracking" so there memory usage stays low. Complete DB Schema importet, no changes on the definition.

  • With an foreach() on all source rows to convert all records

  • It are basically customer specific data converstions(import) that are too complex to do with xml. The 'ORM' Mapper is just used as a more compfortable & discoverable DbDataReader.

Entity Framework seems good for that use case. Good idea to do the same with NHibernate + some generator?

Ayende Rahien
07/30/2010 09:38 AM by
Ayende Rahien

Daniel,

That is when you use NHibernate's Stateless Session

Comments have been closed on this topic.