http://ayende.comAyende @ RahienCopyright (C) Ayende Rahien 2004 - 2021 (c) 202660Mike, You seem to be missing the point. It isn't the time to update the screen that is meaningful. It is the time to process the messages. I'll have a separate post about it, but let us just say that the same problem exists with no UI as well http://ayende.com/4206/how-to-lead-a-convoy-to-safety#comment14http://ayende.com/4206/how-to-lead-a-convoy-to-safety#comment14Mon, 21 Sep 2009 17:43:03 GMTThomas, One of the design goals is to have as little impact as possible on the profiled application. Stopping the profiled application is not an option. http://ayende.com/4206/how-to-lead-a-convoy-to-safety#comment13http://ayende.com/4206/how-to-lead-a-convoy-to-safety#comment13Mon, 21 Sep 2009 17:42:05 GMTCan you gain efficiency through batching? For instance, are you updating the screen on every update? With a slow resource such as a UI, file, or socket, batching can give you better throughput by merging updates and limiting the number of slow calls required. For Example: public void OnBatch(List <updates updates){ ApplyAll(updates); UpdateScreen(); } This way updates are efficiently throttled and the Queue doesn't fall far behind. Of course, this is something that Retlang does for you. [http://code.google.com/p/retlang/](http://code.google.com/p/retlang/)> Mike http://ayende.com/4206/how-to-lead-a-convoy-to-safety#comment12http://ayende.com/4206/how-to-lead-a-convoy-to-safety#comment12Mon, 21 Sep 2009 15:07:51 GMTInstead of dropping messages when you reach a threshold... why not simply block the host application, so it has to wait until it can write the next message to the queue? granted, this would reduce the performance of the host application, but if i want to debug/trace my application i usually would want to get all messages, even if it means that my application may run a bit slower while being traced... http://ayende.com/4206/how-to-lead-a-convoy-to-safety#comment11http://ayende.com/4206/how-to-lead-a-convoy-to-safety#comment11Mon, 21 Sep 2009 14:07:59 GMTHm, I wonder if you could do a meta-analysis over a given number of messages knowing that some messages have been dropped. For example, if your profiler could run, say, 10 times on the same system with approximately the same load, you could average together the results, in a sense, to guarantee a stable conclusion. This would probably require some kind of ability to drop pseudo-random messages though, as you wouldn't be able to rely on just dropping when it starts to get overloaded - if you tried that, then you could very well be missing the exact thing which is causing the overload. Differently, you could define certain messages (and that which they are dependent on) to be knowingly serializable, then only serialize those with a marker saying where they show up in the queue. This would probably end up creating a scheduling problem over the queue, though, so it's most likely not worth it. http://ayende.com/4206/how-to-lead-a-convoy-to-safety#comment10http://ayende.com/4206/how-to-lead-a-convoy-to-safety#comment10Sun, 20 Sep 2009 16:11:45 GMTFrank, It is possible that this would lead to an Out Of Memory Exception And in general it is better not to try walking that line http://ayende.com/4206/how-to-lead-a-convoy-to-safety#comment9http://ayende.com/4206/how-to-lead-a-convoy-to-safety#comment9Sun, 20 Sep 2009 10:43:47 GMTI didn't think UI was the problem...so I gather that the queuing of messages is absolutely "dumb" in that all possible messages are gathered, while I thought that there might be some form of "pre-processing". I suppose that isn't really possible, though, since defining whether a message is "severe" or not probably involves quite a bit of knowledge (= processor time). Otoh, how expensive is RAM these days? If you're profiling an app with such throughput I'd hope that people could spare a few dollars on a couple of GBs. http://ayende.com/4206/how-to-lead-a-convoy-to-safety#comment8http://ayende.com/4206/how-to-lead-a-convoy-to-safety#comment8Sun, 20 Sep 2009 10:28:57 GMTFrank, The problem isn't with showing the information, the problem is in processing it fast enough http://ayende.com/4206/how-to-lead-a-convoy-to-safety#comment7http://ayende.com/4206/how-to-lead-a-convoy-to-safety#comment7Sun, 20 Sep 2009 09:50:57 GMTRafal, InitializeOfflineProfiling() - it is there. :-) http://ayende.com/4206/how-to-lead-a-convoy-to-safety#comment6http://ayende.com/4206/how-to-lead-a-convoy-to-safety#comment6Sun, 20 Sep 2009 09:49:43 GMTRichard, I am sorry, but we have different definitions for what sustained heavy load _means_. When I am talking about this I am talking about doing this for 30 minutes or so of non stop activity. That is rarely the case. Anyway, I already have a branch where I am taking care of this, and I'll publish it sometimes this week. http://ayende.com/4206/how-to-lead-a-convoy-to-safety#comment5http://ayende.com/4206/how-to-lead-a-convoy-to-safety#comment5Sun, 20 Sep 2009 09:47:58 GMTMy question would be...what questions regarding NH usage can NH Prof answer in a heavy load scenario that couldn't be answered when running the app under less heavy load? In such a case it might be OK to have NH Prof "degrade" to processing only messages of severe importance until it catches up again... Of course this falls down again if the application is so bitchy that all messages are severe... http://ayende.com/4206/how-to-lead-a-convoy-to-safety#comment4http://ayende.com/4206/how-to-lead-a-convoy-to-safety#comment4Sun, 20 Sep 2009 09:46:12 GMTMaybe you should add an option of offline profiling - some small component would write all the trace information to a log and NH Prof would then be used to analyze that log? Live profiling is a problem in production environment - if you have memory/performance problems and want to analyze that with a profiler, the profiler will add more load to the system and seriously worsen the situation. http://ayende.com/4206/how-to-lead-a-convoy-to-safety#comment3http://ayende.com/4206/how-to-lead-a-convoy-to-safety#comment3Sun, 20 Sep 2009 09:40:45 GMT"not a realistic scenario for NH Prof" <-- I think you overestimate your customers. I can think of at least half a dozen pages in one web application I work on that take anything from 70-700 SQL/cache requests per hit (30-40 mapped classes, 500 tables, 30GB database). During this time NH Prof frequently becomes unresponsive, and often remains busy for a few secs after the session ended. We know our code is not the best -- using domain models for building a report, automapper resolvers getting more details per item, recursive trees, leaning far too much on the cache etc. Even after lots of fetching/joins/caching tuning there is still lots of SELECT N+1. So unfortunately overloading NH Prof is a very realistic scenario for us. http://ayende.com/4206/how-to-lead-a-convoy-to-safety#comment2http://ayende.com/4206/how-to-lead-a-convoy-to-safety#comment2Sun, 20 Sep 2009 08:42:29 GMTUse an object database :) http://ayende.com/4206/how-to-lead-a-convoy-to-safety#comment1http://ayende.com/4206/how-to-lead-a-convoy-to-safety#comment1Sun, 20 Sep 2009 05:33:09 GMT