<< NHibernate - <natural-id/> | Home | NH Prof: Exposing the data that you already have >>

NH Prof: Getting big, and bigger

Well, at least now I have proof positive that all the architecture work that we did in order to ensure that NH Prof will scale better actually works. The following screenshot shows an example that used to crash NH Prof, it loading an 80 MB file with over 200,000 individual events to scan and process.

image

The fun part is that this is run through the entire pipeline, but without involving the UI, so we get the ability to process all those events without having to wait for the UI. If you remember, we had to invest a lot of time making sure that this sort of scenario will work properly.

Now, it takes 85 seconds for this to load, which I consider pretty reasonable, it means about 2,350 events per second, which isn’t a high number, but is plenty fast enough overall, considering the amount of work that I have to do in order to do things like calculate alerts, format the SQL, and do all the other things that NH Prof has to do.

Print | posted on Thursday, June 25, 2009 10:00 AM

Feedback

Gravatar

# re: NH Prof: Getting big, and bigger 6/25/2009 10:55 AM Rafal

Over 2000 events per second? Impressive, is your queue running on Rhino Queues or sth else? (I'm asking because some time ago I had problems with RQ performance, it was nowhere near 2000 messages/second)

Gravatar

# re: NH Prof: Getting big, and bigger 6/25/2009 11:01 AM Ayende Rahien

Rafal,
No, it is an in memory queue.
And it is actually about the processing time & IO than anything related to the queue.
In memory queue, naively implemented, can easily do millions per second.

I intent to sit on the RQ perf issue when I have free time, mind you.

Gravatar

# re: NH Prof: Getting big, and bigger 6/25/2009 4:12 PM Thomas Krause

How well does the performance scale with the number of cpu cores?

Do you use multi threading at all for processing the events and if yes how do you partition the work between multiple threads?

Gravatar

# re: NH Prof: Getting big, and bigger 6/25/2009 4:40 PM Ayende Rahien

Thomas,
No, processing events is done in a single background thread, that significantly simplify what I need to do to bring it all into a single cohesive model.

Gravatar

# re: NH Prof: Getting big, and bigger 6/26/2009 6:59 AM Fabio Maulo

The "format the SQL" is a responsibility of which tear of NHProf ?

Gravatar

# re: NH Prof: Getting big, and bigger 6/26/2009 8:36 AM Ayende Rahien

Fabio,
This is done on the backend portion of NH Prof

Gravatar

# re: NH Prof: Getting big, and bigger 6/26/2009 8:46 AM Fabio Maulo

If you think that the SQL formatting is a possible point to improve performances perhaps you should move it up, if the SQL formatting is really only to show a fashion SQL to the user.

Gravatar

# re: NH Prof: Getting big, and bigger 6/26/2009 8:52 AM Ayende Rahien

Unfortunately, I actually do need to process all / most of them.
The SQL Formatter part is also responsible for creating the nice listing in the headers, which is almost always needed anyway

Comments have been closed on this topic.