UberProf
UberProf
Hibernate Profiler New Feature: Parameters Values
One of the annoying things about the Hibernate port of the profiler was that JDBC didn’t provide us with the parameters values. Eric has just fixed and that is now live: Enjoy…
Profiler new feature: Too many joins detection
This is Josh’s feature, since we wrote most of the code for it together. Basically, it recognize a very common performance problem, queries that uses too many joins, such as this one: Which would result in the following warning: Queries with too many joins might be a performance problem. Each join requires the database to perform additional work, and the complexity and cost of the query grows rapidly with each additional join. While relational database are optimized for handling joins, it is often more efficient to perform several separate queries...
Profiler Subscription?
One repeated request for the profiler is to have a personal version (which a price to match). I am not really happy with the idea, for several reasons. One of the major ones is that I have enough variability in the product already, and adding a new edition in addition to the 4 we already support is bound to create headaches. Another is that I simply cannot just draw a line and say “those are the pro features and these are the personal features”. Instead, I wonder about offering a subscription model, something with a cost around 10...
Profiler Speculative Feature: Query plans
This isn’t a new feature, because you can’t use it right now, but it is a really nice feature that we are working on, and I couldn’t resist showing it off hot “off the press”, so to speak. Given the following query: SELECT this_.id AS id7_1_, this_.title AS title7_1_, this_.subtitle AS subtitle7_1_, this_.allowscomments AS allowsco4_7_1_, this_.createdat AS createdat7_1_, posts2_.blogid AS blogid3_, posts2_.id AS id3_, ...
Profiler new feature: Integrating with application frameworks
One of the things that makes working with the profiler easier is the fact that it gives you not just information, but information in context. I was working with an app using Rhino Service Bus, and it really bothered me that I couldn’t immediately figure out what was the trigger for a session. When using ASP.Net or WCF, the profiler can show the URL that triggered the request, but when we are not using a url based mechanism, that turns out to be much harder. So I set out to fix that, you can see the results...
Say hello to Uber Prof
I got several requests for this, so I am making Uber Prof itself available for purchasing. What is Uber Prof? It is a short hand way of saying: All the OR/M profilers that we make. An Uber Prof license gives you the ability to use: NHibernate Profiler Hibernate Profiler Linq to SQL Profiler Entity Framework Profiler And it will automatically give you the ability to use any additional profilers that we will create. And yes, there is an upgrade path if...
Profiler New Feature: Side by Side diff
The profiler could do session diffs (showing the difference between executed statements between two sessions) for a while now, but we got some requests for changing it to follow a more traditional source control diff style. This is now done, and it should make it easier to understand the changes between two sessions:
UberProf new feature: Query Plan Cache Misuse
This is a new feature available for NHibernate Profiler*, Linq to SQL Profiler and Entity Profiler. Basically, it detects when the same query is executed with different parameter sizes, which generate different query plan in the query cache. Let us say that we issue two queries, to find users by name. (Note that I am using a syntax that will show you the size of the parameters, to demonstrate the problem). We can do this using the following queries. exec sp_executesql
N'SELECT * FROM Users WHERE Username...
Vote for the UberProf case study in Mix 2010
Christopher Bennage has submitted a Mix talk that I think is interesting :-) LinqToSQL and EntityFramework Profilers: Case Study If you aren’t already familiar with the UberProf suite of ORM profilers, you can read tales of the development on Ayende’s blog. Rob and I built the UI side of the application, and we learned a lot in the process. I’d like to do a talk were we discuss the challenges of the project, how we solved them, and what we did wrong. Yes, NHProf will be included too. (I submitted a case...
The operation was successful, but the patient is still dead… deferring the obvious doesn’t work
So, I have a problem with the profiler. At the root of things, the profiler is managing a bunch of strings (SQL statements, stack traces, alerts, etc). When you start pouring large amount of information into the profiler, the number of strings that it is going to keep in memory is going to increase, until you get to say hello to OutOfMemoryException. During my attempt to resolve this issue, I figured out that string interning was likely to be the most efficient way to resolve my problem. After all, most of the strings that I have to display are...
UberProf performance improvements, nothing helps if you are stupid
The following change took a while to figure out, but it was a huge performance benefit (think, 5 orders of magnitude). The code started as: private readonly Regex startOfParametersSection =
new Regex(@"(;\s*)[@:?]p0 =", RegexOptions.Compiled);
UberProf performance improvements, beware of linq query evaluation
This is a diff from the performance improvement effort of UberProf. The simple addition of .ToList() has significantly improved the performance of this function: Why? Before adding the ToList(), each time we try to run our aggregation functions on the statements enumerable, we would force re-evaluation of the filtering (which can be quite expensive). By adding ToList() I am now making the filtering run only once. There is another pretty obvious performance optimization that can be done here, can you see it? And why did I choose not to implement it?
UberProf performance improvements, or: when O(N^3 + N) is not fast enough
While working on improving the performance of the profiler, I got really annoyed. The UI times for updating sessions with large amount of statements was simply unacceptable. We already virtualized the list UI, so it couldn’t be that. I started tracking it down, and it finally came down to the following piece of code: protected override void Update(IEnumerable<IStatementSnapshot> snapshots)
{
foreach (var statementSnapshot in snapshots)
{
var found = (from model in unfilteredStatements
...
UberProf performance improvements
I mentioned before that I run into some performance problems, I thought it would be interesting to see how I solved them. The underlying theme was finding O(N) operations are eradicating them. In almost all the cases, the code is more complex, but significantly faster Old code New code public static class...
ÜberProf new feature: Fully keyboard enabled
For a long time, most of the work in the profiler (NH Prof, HProf, L2S Prof & EF Prof) could be done only with the use of the mouse. That annoyed a bunch of people, but it didn’t really bother me. Rob fixed this recently, and I cannot believe what kind of a difference it makes. Here are the shortcuts: S Focus on Sessions tab header ...
UberProf performance challenges
One of the things that the profiler is supposed to do is to handle large amount of data, so I was pretty bummed to hear that users are running into issues when they try to use the profiler in high load scenarios, such as load tests. Luckily, one of the things that the profiler can do is output to a file, which let me simulate what is going on on the customer site very easily. The first performance problem is that the profiler isn’t processing things fast enough: The second is that there are...
ÜberProf new feature: Programmatic Integration
Well… I am still working done the list of stuff people request for the profiler (NH Prof, HProf, L2S Prof & EF Prof) , and one of the things that popped into the head of the list was wanting to have programmatic access to the profiler output. We aren’t talking about just the XML reports that are available, but to be able to get the data in a way that is easy to work with. Well, here it is: There is a new DLL in the profiler distribution, HibernatingRhinos.Profiler.Integration, where you can find the CaptureProfilerOutput...
NH Prof new feature: Filter static files
One of the most annoying things about doing the TekPub episodes is finding the right session, because NH Prof will detect any session opening. This means that it usually looks like this: Which also means that it is pretty hard to figure out. I got annoyed enough with that to add a specific filter to filter all of those in one shot (you could already do it, but it meant creating several filters): Which results in a much better experience:
ÜberProf and Continuous Integration
One of the features that keep popping up for ÜberProf is that people want to use that in CI scenarios, usually to be able to programmatically check that they don’t have things like SELECT N+1 popping up, etc. With build 562 of ÜberProf, this is now possible. How does this works? We now have a command line interface for ÜberProf, with the following options: /CmdLineMode[+|-] (short form /C)/File:<string> (short form /F)/ReportFormat:{Xml|Html} (short form...
Advertisement appreciation day
I am trying to get some advertising to get people into Hibernate Profiler. And since I am doing most of the profiler work on the open, I thought I might as well put out the advertising as well. Thanks so much for everyone who helped me out on twitter, but especially to Mark Nijhof, Glenn F. Henriksen and Ian Battersby. Thoughts?
Uber Prof analysis pipeline
In most situations, I like to think about my apps as a set of composite Lego blocks. Ignoring the infrastructure bits, here is how Uber Prof is structured internally:
I have a set of message processors, which work of low level event messages, detect patterns in those messages and turn them into high level events. Next, I have the Model Building part, which takes the high level information and handle correlation between the different events. I then have analyzers, which are branched out of the model building processor to perform the various analysis actions that I need.
The idea is to...
Some observations on Linq to Sql & Entity Framework codebases
So, I have to do a lot of L2S and EF work recently, while trying to create L2SProf and EFProf. The interesting tidbit is that I got a totally different approach to using them than almost anyone else. I don’t really care for either as an OR/M (I have my own, thank you very much), I only care for plugging into them deeply enough that I can get the information that I want. With L2S, that was fairly easy, all I needed to do was figure out how to get access to the logging that it was already...