Exactly 4 years after the 1.0 release of Uber Prof, we have the 2.0 release for Uber Prof.

You can find it here: http://hibernatingrhinos.com/products/

New features include production and cloud profiling, performance improvement and better error detection & guidance.

Well, as the year draws to a close, it is that time again, I got older, apparently. Yesterday marked my 31th trip around the sun.

To celebrate, I decided to give the first 31 people a 31% discount for all of our products.

This offer applies to:

• Raven DB (Standard edition)
• NHibernate Profiler
• Entity Framework Profiler
• Linq to SQL Profiler
• LLBLGen Profiler

This also applies to our support & consulting services.

All you have to do is to use the following coupon code: goodbye-2012

Enjoy the end of the year, and happy holidays.

With Uber Prof 2.0 (NHibernate Profiler, Entity Framework Profiler, Linq to SQL Profiler, LLBLGen Profiler) we are going to bring you a new era of goodness.

In 1.0, we gave you a LOT of goodness for the development stage of building your application, but now we are able to take it a few steps further. Uber Prof 2.0 supports production profiling, which means that you can run it in production and see what is going on in your application now!

To make things even more interesting, we have also done a lot of work to make sure that this works on the cloud as well. For example, go ahead and look at this site: http://efprof.cloudapp.net/

This is a live application, that doesn’t really do anything special, I’ll admit. But the kicker is when you go to this URL: http://efprof.cloudapp.net/profiler/profiler.html

This is EF Prof, running on the cloud, and giving you the results that you want, live. You can read all about this feature and how to enable it here, but I am sure that you can see the implications.

Well, it took a while, but the next version of the NHibernate Profiler is ready to go to a private beta.

We go to private beta before the product is done, because we want to get as much early feedback as we can.

We have 10 spots for NHibernate Profiler v2.0 and 10 spots for Entity Framework Profiler v2.0.

If you are interested, please send me an email about this.

We have been working on the profilers (NHibernate Profiler, Entity Framework Profiler, Linq to SQL Profiler, LLBLGen Profiler and Hibernate Profiler) for close to three years now. And we have been running always as 1.x, so we didn’t have a major release (although we have continual releases, we currently have close to 900 drops of the 1.x version).

The question now becomes, what is going to happen in the next version of the profiler?

• Production Profiling, the ability to setup your application so that you can connect to your production application and see what is going on right now.
• Error Analysis, the ability to provide you with additional insight and common solution to recurring problems.
• Global Query Analysis, the ability to take all of your queries, look at their query plans and show your potential issues.

Those are the big ones, we have a few others, and a surprise in store

What would you want to see in the next version of the profiler?

Fitzchak has all the details in our company blog.

Because I keep getting asked, this feature is available for the following profilers:

• NHibernate Profiler
• Entity Framework Profiler
• Linq to SQL Profiler
• LLBLGen Profiler
• Hibernate Profiler

This new feature detects a very interesting bad practice, write to the database from multiple session in the same web request.

For example, consider the following code:

public void SaveAccount(Account account)
{
    using(var session = sessionFactory.OpenSession())
    using(session.BeginTransaction())
    {
           session.SaveOrUpdate(account);
           session.Transaction.Commit();    
    }
}

public Account GetAccount(int id)
{
    using(var session = sessionFactory.OpenSession())
    {
        return session.Get<Account>(id);
    }
}

It is bad for several reasons, micro managing the session is just one of them, but the worst part is yet to come…

public void MakePayment(int fromAccount, int toAccount, decimal ammount)
{
    var from = Dao.GetAccount(fromAccount);
    var to = Dao.GetAccount(toAccount);
    from.Total -= amount;
    to.Total += amount;
    Dao.SaveAccount(from);
    Dao.SaveAccount(to);
}

Do you see the error here? There are actually several, let me count them:

• We are using 4 different connections to the database in a single method.
• We don’t have transactional safety!!!!

Think about it, if the server crashed between the fifth and sixth lines of this method, where would we be?

We would be in that wonderful land where money disappear into thin air and we stare at that lovely lawsuit folder and then jump from a high window to a stormy sea.

Or, of course, you could use the profiler, which will tell you that you are doing something which should be avoided:

Isn’t that better than swimming with the sharks?

Recently, we added a way to track alerts across all the sessions the request. This alert will detect whenever you are making too many database calls in the same request.

But wait, don’t we already have that?

Yes, we do, but that was limited to the scope of one session. there is a very large set of codebases where the usage of OR/Ms is… suboptimal (in other words, they could take the most advantage of the profiler abilities to detect issues and suggest solutions to them), but because of the way they are structured, they weren’t previously detected.

What is the difference between a session and a request?

Note: I am using NHibernate terms here, but naturally this feature is shared among all profiler:

• NHibernate Profiler
• Entity Framework Profiler
• Linq to SQL Profiler
• LLBLGen Profiler
• Hibernate Profiler

A session is the NHibernate session (or the data/object context in linq to sql / entity framework), and the request is the HTTP request or the WCF operation. If you had code such as the following:

public T GetEntity<T>(int id)
{
    using (var session = sessionFactory.OpenSession())
    {
         return session.Get<T>(id);
    }
}

This code is bad, it micro manages the session, it uses too many connections to the database, it … well, you get the point. The problem is that code that uses this code:

public IEnumerable<Friends> GetFriends(int[] friends)
{
   var results = new List<Friends>();
   foreach(var id in friends)
       results.Add(GetEnttiy<Friend>(id));

   return results;
}

The code above would look like the following in the profiler:

As you can see, each call is in a separate session, and previously, we wouldn’t have been able to detect that you have too many calls (because each call is a separate session).

Now, however, we will alert the user with a too many database calls in the same request alerts.

We have recently been doing some work on Uber Prof, mostly in the sense of a code review, and I wanted to demonstrate how easy it was to add a new feature. The problem is that we couldn’t really think of a nice feature to add that we didn’t already have.

Then we started thinking about features that aren’t there and that there wasn’t anything in Uber Prof to enable, and we reached the conclusion that one limitation we have right now is the inability to analyze your application’s behavior beyond the session’s level. But there is actually a whole set of bad practices that are there when you are using multiple sessions.

That led to the creation of a new concept the Cross Session Alert, unlike the alerts we had so far, those alerts looks at the data stream with a much broader scope, and they can analyze and detect issues that we previously couldn’t detect.

I am going to be posting extensively on some of the new features in just a bit, but in the meantime, why don’t you tell me what sort of features do you think this new concept is enabling.

And just a reminder, my architecture is based around Concepts & Features.

Originally posted at 1/7/2011

Because I keep getting asked, this feature is available for the following profilers:

• NHibernate Profiler
• Entity Framework Profiler
• Linq to SQL Profiler
• LLBLGen Profiler
• Hibernate Profiler

This feature is actually two separate ones. The first is the profiler detecting what is the most expensive part of the query plan and making it instantly visible. As you can see, in this fairly complex query, it is this select statement that is the hot spot.

Another interesting feature that only crops up whenever we are dealing with complex query plans is that the query plan can get big. And by that I mean really big. Too big for a single screen.

Therefore, we added zooming capabilities as well as the mini map that you see in the top right corner.

Originally posted at 1/7/2011

This is another oft requested feature that we just implemented. The new feature is available for the full suite of Uber Profilers:

• NHibernate Profiler
• Entity Framework Profiler
• Linq to SQL Profiler
• LLBLGen Profiler
• Hibernate Profiler

You can see the new feature below:

I think it is cute, and was surprisingly easy to do.

Uber Prof have recently passed the stage where it is mostly implemented using itself, so I just had to wire a few things together, and then I spent most of the time just making sure that things aligned correctly on the UI.

To celebrate the new year, I decided to offer a single day coupon that will get you 35% discount for all my profiler products.

The coupon code is valid until the 1st of January 2011, but I won’t mention in which timezone, so you might want to hurry up.

The coupon code is:

HNY-45K2D465DD

You can use it to buy:

• NHibernate Profiler
• Entity Framework Profiler
• Linq to SQL Profiler
• LLBLGen Profiler
• Hibernate Profiler

Happy new year!

Originally posted at 11/25/2010

In a recent post, I discussed the notion of competitive advantage and how you should play around them. In this post, I am going to focus on Uber Prof. Just to clarify, when I am talking about Uber Prof, I am talking about NHibernate Profiler, Entity Framework Profiler, Linq to SQL Profiler, Hibernate Profiler and LLBLGen Profiler. Uber Prof is just a handle for me to use to talk about each of those.

So, what is the major competitive advantage that I see in the Uber Prof line of products?

Put very simply, they focus very heavily on the developer’s point of view.

Other profilers will give you the SQL that is being executed, but Uber Prof will show you the SQL and:

• Format that SQL in a way that make it easy to read.
• Group the SQL statements into sessions. Which let the developer look at what is going on in the natural boundary.
• Associate each query with the exact line of code that executed it.
• Provide the developer with guidance about improving their code.

There are other stuff, of course, but those are the core features that make Uber Prof into what it is.

I have been doing some studying of how people are using the profiler, and it shows some interesting results.

• Typical profiler session is :
• NH Prof : 1:15 hours
• Hibernate Profiler: 1:05 hours
• EF Prof: 42 minutes
• L2S Prof: 50 minutes
• 83% of the profiler users have used it more than once. In fact, here is the # of usages:
  
  So we have over 50% that use it regularly.
• Most people use it predominately to view the statements executed:
  
  This means that the reports are getting comparatively little attention.
• The results per geographical location are also interesting:
  

One of the annoying things about the Hibernate port of the profiler was that JDBC didn’t provide us with the parameters values.

Eric has just fixed and that is now live:

Enjoy…

This is Josh’s feature, since we wrote most of the code for it together. Basically, it recognize a very common performance problem, queries that uses too many joins, such as this one:

Which would result in the following warning:

Queries with too many joins might be a performance problem. Each join requires the database to perform additional work, and the complexity and cost of the query grows rapidly with each additional join. While relational database are optimized for handling joins, it is often more efficient to perform several separate queries instead of a single query with several joins in it.

For OLTP systems, you should consider simplifying your queries or simplifying the data model. While I do not recommend avoiding joins completely, I strong discourage queries with large numbers of joins. Another issue to pay attention to is possible Cartesian products in queries contains joins, it is very easy to create such a thing and not notice it during development.

One repeated request for the profiler is to have a personal version (which a price to match).

I am not really happy with the idea, for several reasons. One of the major ones is that I have enough variability in the product already, and adding a new edition in addition to the 4 we already support is bound to create headaches. Another is that I simply cannot just draw a line and say “those are the pro features and these are the personal features”.

Instead, I wonder about offering a subscription model, something with a cost around 10 – 15 Euro per month. This would be renewed monthly (automatically), and beyond just having a lower price point, it will also provide automatic upgrades across versions (so free upgrade from 1.x to 2.x).

Thoughts?

This isn’t a new feature, because you can’t use it right now, but it is a really nice feature that we are working on, and I couldn’t resist showing it off hot “off the press”, so to speak.

Given the following query:

SELECT this_.id             AS id7_1_,
       this_.title          AS title7_1_,
       this_.subtitle       AS subtitle7_1_,
       this_.allowscomments AS allowsco4_7_1_,
       this_.createdat      AS createdat7_1_,
       posts2_.blogid       AS blogid3_,
       posts2_.id           AS id3_,
       posts2_.id           AS id0_0_,
       posts2_.title        AS title0_0_,
       posts2_.TEXT         AS text0_0_,
       posts2_.postedat     AS postedat0_0_,
       posts2_.blogid       AS blogid0_0_,
       posts2_.userid       AS userid0_0_
FROM   blogs this_
       LEFT OUTER JOIN posts posts2_
         ON this_.id = posts2_.blogid
WHERE  this_.id = 1 /* @p0 */

SELECT this_.id            AS id0_1_,
       this_.title         AS title0_1_,
       this_.TEXT          AS text0_1_,
       this_.postedat      AS postedat0_1_,
       this_.blogid        AS blogid0_1_,
       this_.userid        AS userid0_1_,
       comments2_.postid   AS postid3_,
       comments2_.id       AS id3_,
       comments2_.id       AS id2_0_,
       comments2_.name     AS name2_0_,
       comments2_.email    AS email2_0_,
       comments2_.homepage AS homepage2_0_,
       comments2_.ip       AS ip2_0_,
       comments2_.TEXT     AS text2_0_,
       comments2_.postid   AS postid2_0_
FROM   posts this_
       LEFT OUTER JOIN comments comments2_
         ON this_.id = comments2_.postid
WHERE  this_.blogid = 1 /* @p1 */

The profiler can show you the query plan using this UI:

And here is how the same query looks like using the query plan feature in Management Studio:

So, why implement it?

• This isn’t limited to SQL Server, the profiler can display query plans for: SQL Server, Oracle, PostgreSQL and MySQL
• This let you keep yourself in the flow, just hit a button to see the query plan, instead of copying the SQL, opening SSMS, displaying the query plan, etc.

Don’t discount the last one, making it easy is one of the core values of the profiler.

The idea is that if you make it easy enough, the barriers for using it goes away. If you can instantly see the query plan for a query, you are far more likely to look at it than if it takes 30 seconds to get that. At that point, you would only do it when you already have a performance problem.

One of the things that makes working with the profiler easier is the fact that it gives you not just information, but information in context.

I was working with an app using Rhino Service Bus, and it really bothered me that I couldn’t immediately figure out what was the trigger for a session. When using ASP.Net or WCF, the profiler can show the URL that triggered the request, but when we are not using a url based mechanism, that turns out to be much harder.

So I set out to fix that, you can see the results below:

This session was generated by a message batch containing messages for MyBooks, MyQueue, etc.

The integration is composed of two parts, first, from the profiler perspective, you now have the ProfilerIntegration.CurrentSessionContext property, which allows you to customize how the profiler detects the current context.

The second part is the integration from the application framework itself, you can see how I did that for Rhino Service Bus, which will dynamically detect the presence of the profiler and fill the appropriate values. The result makes it a lot easier to track down what is going on.

I got several requests for this, so I am making Uber Prof itself available for purchasing.

What is Uber Prof?

It is a short hand way of saying: All the OR/M profilers that we make.

An Uber Prof license gives you the ability to use:

• NHibernate Profiler
• Hibernate Profiler
• Linq to SQL Profiler
• Entity Framework Profiler

And it will automatically give you the ability to use any additional profilers that we will create. And yes, there is an upgrade path if you already purchased a single profiler license and would like to upgrade to Uber Prof.

The profiler could do session diffs (showing the difference between executed statements between two sessions) for a while now, but we got some requests for changing it to follow a more traditional source control diff style.

This is now done, and it should make it easier to understand the changes between two sessions:

This is a new feature available for NHibernate Profiler*, Linq to SQL Profiler and Entity Profiler. Basically, it detects when the same query is executed with different parameter sizes, which generate different query plan in the query cache.

Let us say that we issue two queries, to find users by name. (Note that I am using a syntax that will show you the size of the parameters, to demonstrate the problem).

We can do this using the following queries.

exec sp_executesql 
      N'SELECT * FROM Users WHERE Username = @username',
      N'@username nvarchar(3)',
      @username=N'bob'
exec sp_executesql 
      N'SELECT * FROM Users WHERE Username = @username',
      N'@username nvarchar(4)',
      @username=N'john'

This sort of code result in two query plans stored in the database query cache, because of the different parameter sizes. In fact, if we assume that the Username column has a length of 16, this single query may take up 16 places in the query cache.

Worse, if you have two parameters whose size change, such as username (length 16) and password (length 16), you may take up to 256 places in the query cache. Obviously, if you use more parameters, or if their length is higher, the number of places that a single query can take in the query cache goes up rapidly.

This can cause performance problems as the database need to keep track of more query plans (uses more memory) may need evict query plans from the cache, which would result in having to rebuild the query plan (increase server load and query time).

* Please note that detecting this in NHibernate requires the trunk version of NHibernate. And it is pretty useless there, since on the trunk, NHibernate will never generate this issue.

Christopher Bennage has submitted a Mix talk that I think is interesting :-)

LinqToSQL and EntityFramework Profilers: Case Study

If you aren’t already familiar with the UberProf suite of ORM profilers, you can read tales of the development on Ayende’s blog. Rob and I built the UI side of the application, and we learned a lot in the process. I’d like to do a talk were we discuss the challenges of the project, how we solved them, and what we did wrong.

Yes, NHProf will be included too. (I submitted a case study for it last year, and it didn’t get picked. I have to sneak it in).

A few interesting aspects:

• we built this using MVVM, but well before Caliburn reached maturity.
• the four separate apps (NHProf, EFProf, L2SProf, HProf) all use a single code base.
• we’re about to port the project from WPF to Silverlight.

Please vote for this session.

So, I have a problem with the profiler. At the root of things, the profiler is managing a bunch of strings (SQL statements, stack traces, alerts, etc). When you start pouring large amount of information into the profiler, the number of strings that it is going to keep in memory is going to increase, until you get to say hello to OutOfMemoryException.

During my attempt to resolve this issue, I figured out that string interning was likely to be the most efficient way to resolve my problem. After all, most of the strings that I have to display are repetitive. String interning has one problem, it exists forever. I spent a few minutes creating a garbage collectible method of doing string interning. In my first test, which was focused on just interning stack traces, I was able to reduce memory consumption by 50% (about 800Mb, post GC) and it is fully garbage collectible, so it won’t hung around forever.

Sounds good, right?

Well, not really. While it is an interesting thought experiment, using interning is a great way of handling things, but it only mask the problem, and that only for a short amount of time. The problem is still an open ended set of data that I need to deal with, and while there are a whole bunch of stuff that I can do to delay the inevitable, defeat is pretty much ensured. The proper way of doing that is not trying to use hacks to reduce memory usage, but to deal with the root cause, keeping everything in memory.

The following change took a while to figure out, but it was a huge performance benefit (think, 5 orders of magnitude). The code started as:

private readonly Regex startOfParametersSection = 
            new Regex(@"(;\s*)[@:?]p0 =", RegexOptions.Compiled);

And the optimization is:

private static readonly Regex startOfParametersSection = 
            new Regex(@"(;\s*)[@:?]p0 =", RegexOptions.Compiled);

The story behind this is interesting, this piece of code (and a few others like it) used to be in a class that has a singleton lifestyle. At some point, it was refactored into a command class that is created often, which obviously had… drastic effect on the system performance.