Ayende @ Rahien

Refunds available at head office

NH Prof: Getting big, and bigger

Well, at least now I have proof positive that all the architecture work that we did in order to ensure that NH Prof will scale better actually works. The following screenshot shows an example that used to crash NH Prof, it loading an 80 MB file with over 200,000 individual events to scan and process.

image

The fun part is that this is run through the entire pipeline, but without involving the UI, so we get the ability to process all those events without having to wait for the UI. If you remember, we had to invest a lot of time making sure that this sort of scenario will work properly.

Now, it takes 85 seconds for this to load, which I consider pretty reasonable, it means about 2,350 events per second, which isn’t a high number, but is plenty fast enough overall, considering the amount of work that I have to do in order to do things like calculate alerts, format the SQL, and do all the other things that NH Prof has to do.

Comments

Rafal
06/25/2009 07:55 AM by
Rafal

Over 2000 events per second? Impressive, is your queue running on Rhino Queues or sth else? (I'm asking because some time ago I had problems with RQ performance, it was nowhere near 2000 messages/second)

Ayende Rahien
06/25/2009 08:01 AM by
Ayende Rahien

Rafal,

No, it is an in memory queue.

And it is actually about the processing time & IO than anything related to the queue.

In memory queue, naively implemented, can easily do millions per second.

I intent to sit on the RQ perf issue when I have free time, mind you.

Thomas Krause
06/25/2009 01:12 PM by
Thomas Krause

How well does the performance scale with the number of cpu cores?

Do you use multi threading at all for processing the events and if yes how do you partition the work between multiple threads?

Ayende Rahien
06/25/2009 01:40 PM by
Ayende Rahien

Thomas,

No, processing events is done in a single background thread, that significantly simplify what I need to do to bring it all into a single cohesive model.

Fabio Maulo
06/26/2009 03:59 AM by
Fabio Maulo

The "format the SQL" is a responsibility of which tear of NHProf ?

Ayende Rahien
06/26/2009 05:36 AM by
Ayende Rahien

Fabio,

This is done on the backend portion of NH Prof

Fabio Maulo
06/26/2009 05:46 AM by
Fabio Maulo

If you think that the SQL formatting is a possible point to improve performances perhaps you should move it up, if the SQL formatting is really only to show a fashion SQL to the user.

Ayende Rahien
06/26/2009 05:52 AM by
Ayende Rahien

Unfortunately, I actually do need to process all / most of them.

The SQL Formatter part is also responsible for creating the nice listing in the headers, which is almost always needed anyway

Comments have been closed on this topic.