+972 52-548-6969
NH ProfGetting big, and bigger
Well, at least now I have proof positive that all the architecture work that we did in order to ensure that NH Prof will scale better actually works. The following screenshot shows an example that used to crash NH Prof, it loading an 80 MB file with over 200,000 individual events to scan and process.
The fun part is that this is run through the entire pipeline, but without involving the UI, so we get the ability to process all those events without having to wait for the UI. If you remember, we had to invest a lot of time making sure that this sort of scenario will work properly.
Now, it takes 85 seconds for this to load, which I consider pretty reasonable, it means about 2,350 events per second, which isn’t a high number, but is plenty fast enough overall, considering the amount of work that I have to do in order to do things like calculate alerts, format the SQL, and do all the other things that NH Prof has to do.
More posts in "NH Prof" series:
- (09 Dec 2010) Alert on bad ‘like’ query
- (10 Dec 2009) Filter static files
- (16 Nov 2009) Exporting Reports
- (08 Oct 2009) NHibernate Search Integration
- (19 Aug 2009) Multiple Session Factory Support
- (07 Aug 2009) Diffing Sessions
- (06 Aug 2009) Capturing DDL
- (05 Aug 2009) Detect Cross Thread Session Usage
- (22 May 2009) Detecting 2nd cache collection loads
- (15 May 2009) Error Detection
- (12 May 2009) Queries by Url
- (04 Feb 2009) View Query Results
- (18 Jan 2009) Superfluous <many-to-one> update
- (18 Jan 2009) URL tracking
- (10 Jan 2009) Detecting distributed transactions (System.Transactions)
- (06 Jan 2009) The Query Cache
- (05 Jan 2009) Query Duration
- (24 Dec 2008) Unbounded result sets
- (24 Dec 2008) Row Counts
Comments
Over 2000 events per second? Impressive, is your queue running on Rhino Queues or sth else? (I'm asking because some time ago I had problems with RQ performance, it was nowhere near 2000 messages/second)
Rafal,
No, it is an in memory queue.
And it is actually about the processing time & IO than anything related to the queue.
In memory queue, naively implemented, can easily do millions per second.
I intent to sit on the RQ perf issue when I have free time, mind you.
How well does the performance scale with the number of cpu cores?
Do you use multi threading at all for processing the events and if yes how do you partition the work between multiple threads?
Thomas,
No, processing events is done in a single background thread, that significantly simplify what I need to do to bring it all into a single cohesive model.
The "format the SQL" is a responsibility of which tear of NHProf ?
Fabio,
This is done on the backend portion of NH Prof
If you think that the SQL formatting is a possible point to improve performances perhaps you should move it up, if the SQL formatting is really only to show a fashion SQL to the user.
Unfortunately, I actually do need to process all / most of them.
The SQL Formatter part is also responsible for creating the nice listing in the headers, which is almost always needed anyway
Comment preview