Ayende @ Rahien

NHibernate Profiler and Entity Framework Profiler 5.0 RTM

Tue, 17 Apr 2018 09:00:00 GMT

I’m really happy to announce that we have just release a brand new version of NHibernate Profiler and Entity Framework Profiler.

What is new in for NHibernate?

Support for NHibernate 5.1 and 5.1.1
Support for .NET Core

supported on the following platforms: netstandard2.0, net46, netcoreapp2.0

Fixed various minor issues regarding showing duplicate errors and warnings from NHibernate.
Better support for DateTime precision issues in NHibernate 5.0

What is new for Entity Framework:

Support for EF Core

supported on the following platforms: netstandard2.0, net46, netcoreapp2.0
netstandard1.6 is also supported via a separate dll.

Support for DataTable data type in custom reporting
Support for ReadCount and RecordsAFfected in EF Core 2.0
Fixed issue for EF 6 on .NET 4.7
Can report using thread name, not just application name
Provide integration hooks for ASP.Net Core to provide contextual information for queries.

New stuff for both of them:

Improved column type mismatch warning
Support for UniqueIdentifier parameters type
Support for integration with VS 2017.

re: Entity Framework Core performance tuning–part I

Wed, 04 Oct 2017 09:00:00 GMT

I run into a really interesting article about performance optimizations with EF Core and I thought that it deserve a second & third look. You might have noticed that I have been putting a lot of emphasis on performance and I had literally spent years on optimizing relational database access patterns, including building a profiler dedicated for inspecting what an OR/M is doing. I got the source and run the application.

I have a small bet with myself, saying that in any application using a relational database, I’ll be able to find a SELECT N+1 issue within one hour. So far, I think that my rate is 92% or so. In this case, I found the SELECT N+1 issue on the very first page load.

Matching this to the code, we have:

Which leads to:

And here we can already tell that there is a problem, we aren’t accessing the authors. This actually happens here:

So we have the view that is generating 10 out of 12 queries. And the more results per page you have, the more this costs.

But this is easily fixed once you know what you are looking at. Let us look at something else, the actual root query, it looks like this:

Yes, I too needed a minute to recover from this. We have:

One JOIN
Two correlated sub queries

Jon was able to optimize his code by 660ms to 80ms, which is pretty awesome. But that is all by making modifications to the access pattern in the database.

Given what I do for a living, I’m more interested in what it does inside the database, and here is what the query plan tells us:

There are only a few tens of thousands of records and the query is basically a bunch of index seeks and nested loop joins. But note that the way the query is structured forces the database to evaluate all possible results, then filter just the top few. That means that you have to wait until the entire result set has been processed, and as the size of your data grows, so will the cost of this query.

I don’t think that there is much that can be done here, given the relational nature of the data access ( no worries, I’m intending to write another post in this series, you guess what I’m going to write there, right? ).

NHibernate Profiler & Entity Framework Profiler 4.0–Beta Release

Thu, 26 Jan 2017 10:00:00 GMT

I’m happy to announce that the next version of the NHibernate and Entity Framework Profilers is ready for beta testing

You can download the new bits from here.

Entity Framework – Support for EF Core (on .NET Core and .Net 4.6)
NHibernate Profiler – Support for NHibernate 4.1
Persisting interesting sessions across restarts.
Improved filtering of SQL statements and behaviors, persisting your configuration across restarts and generally behaving as a responsible citizen.
The ability to configure profiling via a watched file, to enable admins to enable/disable profiling on the fly.
Allowing you to edit and run a SQL statement as well as compare the database query plan generated so you can get to an optimal solution.
Various usability improvements (auto tracking of interesting data, always on top, compact mode for quick visualizations, better keyboard navigation, etc).

Please take them for a spin and provide us with feedback on it.

Special Offer: 29% discount for all our products

Wed, 27 May 2015 08:16:00 GMT

Well, it is nearly the 29 May, and that means that I have been married for four years.

To celebrate that, I am offering a 29% discount on all our products (RavenDB, NHibernate Profiler, Entity Framework Profiler).

All you have to do is purchase any of our products using the following coupon code:

4th Anniversary

This offer is valid to the end of the month only.

End of year discount, last 2 coupons remaining

Wed, 31 Dec 2014 08:08:00 GMT

Reminding you that the end of year coupon has 2 remaining uses.

You can use: 0x21-celebrate-new-year to purchase any of the following with a 21% discount:

This is valid for today only, so hurry up.

End of year discount coupon has 7 remaining uses

Tue, 23 Dec 2014 14:00:00 GMT

Reminding you that the end of year coupon has 7 remaining uses.

You can use: 0x21-celebrate-new-year to purchase any of:

Happy Holidays and a great new years.

End of year discount for all our products

Thu, 11 Dec 2014 10:00:00 GMT

To celebrate the new year, we offer a 21% discount for all our products. This is available for the first 33 customers that use the coupon code: 0x21-celebrate-new-year

In previous years, we offered a similar number of uses for the coupon code, and they run out fast, so hurry up. This offer is valid for:

Happy Holidays and a great new years.

On a personal note, this marks the full release of all our product lines, and it took an incredible amount of work. I'm very pleased that we have been able to get the new version out there and in your hands, and to have you start making use of the features that we have been working on for so long.

NHibernate & Entity Framework Profiler 3.0

Thu, 04 Dec 2014 10:00:00 GMT

It may not get enough attention, but we have been working on the profilers as well during the past few months.

TLDR; You can get the next generation of NHibernate Profiler and Entity Framework Profiler now, lots of goodies to look at!

I’m sure that a lot of people would be thrilled to hear that we dropped Silverlight in favor of going back to WPF UI. The idea was that we would be able to deploy anywhere, including in production. But Silverlight just made things harder all around, and customers didn’t like the production profiling mode.

Production Profiling

We have changed how we profile in production. You can now make the following call in your code:

NHibernateProfiler.InitializeForProduction(port, password);

And then connect to your production system:

At which point you can profile what is going on in your production system safely and easily. The traffic between your production server and the profiler is SSL encrypted.

NHibernate 4.x and Entity Framework vNext support

The profilers now support the latest version of NHibernate and Entity Framework. That include profiling async operations, better suitability for modern apps, and more.

New SQL Paging Syntax

We are now properly support SQL Server paging syntax:

select * from Users
order by Name
offset 0 /* @p0 */ rows fetch next 250 /* @p1 */ rows only

This is great for NHibernate users, who finally can have a sane paging syntax as well as beautiful queries in the profiler.

At a glance view

A lot of the time, you don’t want the profiler to be front and center, you want to just run it and have it there to glance at once in a while. The new compact view gives you just that:

You can park it at some point in your screen and work normally, glancing to see if it found anything. This is much less distracting than the full profiler for normal operations.

Scopes and groups

When we started working on the profilers, we followed the “one session per request” rule, and that was pretty good. But a lot of people, especially in the Entity Framework group are using multiple sessions or data contexts in a single request, but they still want to see the ability to see the operations in a request at a glance. We are now allowing you to group things, like this:

By default, we use the current request to group things, but we also give you the ability to define your own scopes. So if you are profiling NServiceBus application, you can set the scope as your message handling by setting ProfilerIntegration.GetCurrentScopeName or explicitly calling ProfilerIntegration.StartScope whenever you want.

Customized profiling

You can now surface troublesome issues directly from your code. If you have an issue with a query, you can mark it for attention using CustomQueryReporting .ReportError() that would flag it in the UI for further investigation.

You can also just mark interesting pieces in the UI without an error, like so:

using (var db = new Entities(conStr))
{
    var post1 = db.Posts.FirstOrDefault();

    using (ProfilerIntegration.StarStatements("Blue"))
    {
        var post2 = db.Posts.FirstOrDefault();
    }

    var post3 = db.Posts.FirstOrDefault();
    
    ProfilerIntegration.StarStatements();
    var post4 = db.Posts.FirstOrDefault();
    ProfilerIntegration.StarStatementsClear();

    var post5 = db.Posts.FirstOrDefault();
}

Which will result in:

Disabling profiler from configuration

You can now disable the profiler by setting:

<add key="HibernatingRhinos.Profiler.Appender.NHibernate" value="Disabled" />

This will avoid initializing the profiler, obviously. The intent is that you can setup production profiling, disable it by default, and enable it selectively if you need to actually figure things out.

Odds & ends

We move to WebActivatorEx from the deprecated WebActivator, added xml file for the appender, fixed a whole bunch of small bugs, the most important among them is:

Linq to SQL, Hibernate and LLBLGen Profilers, RIP

You might have noticed that I talked only about NHibernate and Entity Framework Profilers. The sales for the rests weren’t what we hoped they would be, and we are no longer going to sale them.

Go get them, there is a new release discount

You can get the NHibernate Profiler and Entity Framework Profiler for a 15% discount for the next two weeks.

Working on the Uber Profiler 3.0

Wed, 28 May 2014 09:00:00 GMT

One of the things that tend to get lost is the fact that Hibernating Rhinos is doing more than just working on RavenDB. The tagline we like to use is Easier Data, and the main goal we have is to provide users with better ways to access, monitor and manage their data.

We have started planning the next version of the Uber Profiler, that means Entity Framework Profiler, NHibernate Profiler, etc. Obviously, we’ll offer support for EF 7.0 and NHibernate 4.0, and we had gathered quite a big dataset for common errors when using an OR/M, which we intend to convert into useful guidance whenever our users run into such an issue.

But I realized that I was so busy working on RavenDB that I forgot to even mention the work we’ve been doing elsewhere. And I also wanted to solicit feedback about the kind of features you’ll want to see in the 3.0 version of the profilers.

Special Offer: 29% discount for all our products

Sun, 26 May 2013 10:12:00 GMT

Well, it is nearly the 29 May, and that means that I have been married for two years.

To celebrate that, I am offering a 29% discount on all our products (RavenDB, NHibernate Profiler, Entity Framework Profiler, etc).

All you have to do is purchase any of our products using the following coupon code:

2nd anniversary

This offer is valid to the end of the month, so hurry up.

Uber Prof 2.0 is OUT!

Tue, 01 Jan 2013 10:00:00 GMT

Exactly 4 years after the 1.0 release of Uber Prof, we have the 2.0 release for Uber Prof.

You can find it here: http://hibernatingrhinos.com/products/

New features include production and cloud profiling, performance improvement and better error detection & guidance.

Goodbye, 2012: Our end of year discount starts now!

Fri, 21 Dec 2012 05:02:00 GMT

Well, as the year draws to a close, it is that time again, I got older, apparently. Yesterday marked my 31th trip around the sun.

To celebrate, I decided to give the first 31 people a 31% discount for all of our products.

This offer applies to:

This also applies to our support & consulting services.

All you have to do is to use the following coupon code: goodbye-2012

Enjoy the end of the year, and happy holidays.

Production Cloud Profiling With Uber Prof

Tue, 18 Dec 2012 10:00:00 GMT

With Uber Prof 2.0 (NHibernate Profiler, Entity Framework Profiler, Linq to SQL Profiler, LLBLGen Profiler) we are going to bring you a new era of goodness.

In 1.0, we gave you a LOT of goodness for the development stage of building your application, but now we are able to take it a few steps further. Uber Prof 2.0 supports production profiling, which means that you can run it in production and see what is going on in your application now!

To make things even more interesting, we have also done a lot of work to make sure that this works on the cloud as well. For example, go ahead and look at this site: http://efprof.cloudapp.net/

This is a live application, that doesn’t really do anything special, I’ll admit. But the kicker is when you go to this URL: http://efprof.cloudapp.net/profiler/profiler.html

This is EF Prof, running on the cloud, and giving you the results that you want, live. You can read all about this feature and how to enable it here, but I am sure that you can see the implications.

Uber Prof V2.0 is now in Public Beta

Mon, 22 Oct 2012 06:25:00 GMT

Well, we worked quite a bit on that, but the Uber Prof (NHibernate Profiler, Entity Framework Profiler, Linq to SQL Profiler, etc) version 2.0 are now out for public beta.

We made a lot of improvements. Including performance, stability and responsiveness, but probably the most important thing from the user perspective is that we now support running the profiler in production, and even on the cloud.

We will have the full listing of all the new goodies up on the company site soon, including detailed instructions on how to enable production profiling and on cloud profiling, but I just couldn’t wait to break the news to you.

In fact, along with V2.0 of the profilers, we have a brand new site for our company, which you can check here: http://hibernatingrhinos.com/.

To celebrate the fact that we are going on beta, we also offer a 20% discount for the duration of the beta.

Nitpicker corner, please remember that this is a beta, there are bound to be problems, and we will fix them as soon as we can.

Über Profiler v2.0–Private Beta is open

Thu, 16 Aug 2012 09:00:00 GMT

Well, it took a while, but the next version of the NHibernate Profiler is ready to go to a private beta.

We go to private beta before the product is done, because we want to get as much early feedback as we can.

We have 10 spots for NHibernate Profiler v2.0 and 10 spots for Entity Framework Profiler v2.0.

If you are interested, please send me an email about this.

An Entity Framework Profiler user story

Tue, 05 Jun 2012 09:00:00 GMT

One of the things that I really love about build things like the profilers and RavenDB is that it gives me the chance to really spread a lot of my experience in codified form. Occasionally, we get customer feedback to that effect, and I thought that I would share this one with you.

Jeremy Holt have been using the Entity Framework Profiler and has the following to say:

The biggest issue it has highlighted for me is the question of the Context being created on different threads (the N+1 was easy to get my head around and resolve). The problem is the difficulty in finding definitive examples of how/when the Context should be created. (Also found my unbound queries – thanks for that!!)
It was working great (except for a couple of threading issues J) until I ran EFProf. I love the “page of shame” in EFProf – I think you called it “query analysis alerts” or something pompous like that – may I suggest you rename it to “errors dumb programmers make” J

I don’t think that this will be renamed, but this does point out something very important. What we are doing with the profilers is much more than simply show you what your application is doing. We are actually analyzing this data through a complex set of rules in order to find common pitfalls. In Jeremy’s case, the common ones were cross thread usage and Select N+1.

And by highlighting those issues, the profiler make it easy to do so. More so, it make you feel good about doing this, in a way that you wouldn’t get from standard analytics tools.

By having thresholds that you can visibly see and avoid, you get to create better software.

Application analysis: Northwind.NET

Fri, 30 Dec 2011 10:00:00 GMT

For an article I am writing, I wanted to compare a RavenDB model to a relational model, and I stumbled upon the following Northwind.NET project.

I plugged in the Entity Framework Profiler and set out to watch what was going on. To be truthful, I expected it to be bad, but I honestly did not expect what I got. Here is a question, how many queries does it take to render the following screen?

The answer, believe it or no, is 17:

You might have noticed that most of the queries look quite similar, and indeed, they are. We are talking about 16(!) identical queries:

SELECT [Extent1].[ID]           AS [ID],
       [Extent1].[Name]         AS [Name],
       [Extent1].[Description]  AS [Description],
       [Extent1].[Picture]      AS [Picture],
       [Extent1].[RowTimeStamp] AS [RowTimeStamp]
FROM   [dbo].[Category] AS [Extent1]

Looking at the stack trace for one of those queries led me to:

And to this piece of code:

You might note that dynamic is used there, for what reason, I cannot even guess. Just to check, I added a ToArray() to the result of GetEntitySet, and the number of queries dropped from 17 to 2, which is more reasonable. The problem was that we passed an IQueryable to the data binding engine, which ended up evaluating the query multiple times.

And EF Prof actually warns about that, too:

At any rate, I am afraid that this project suffer from similar issues all around, it is actually too bad to serve as the bad example that I intended it to be.

Gilad Shalit is back

Tue, 18 Oct 2011 10:00:00 GMT

I don’t usually do posts about current events, but this one is huge.

To celebrate the event, you can use the following coupon code: SLT-45K2D4692G to get 19.41% discount (Gilad was captive for 1,941 days) for all our profilers:

Hell, even our commercial support for NHibernate is participating.

Please note that any political comment to this post that I don’t agree with will be deleted.

Entity Framework 4.1 Update 1, Backward Compatibility and Microsoft

Fri, 05 Aug 2011 09:00:00 GMT

One of the things that you keep hearing about Microsoft products is how much time and effort is dedicated to ensuring Backward Compatibility. I have had a lot of arguments with Microsoft people about certain design decisions that they have made, and usually the argument ended up with “We have to do it this was to ensure Backward Compatibility”.

That sucked, but at least I could sleep soundly, knowing that if I had software running on version X of Microsoft, I could at least be pretty sure that it would work in X+1.

Until Entity Framework 4.1 Update 1 shipped, that is. Frans Bouma has done a great job in describing what the problem actually is, including going all the way and actually submitting a patch with the code required to fix this issue.

But basically, starting with Entity Framework 4.1 Update 1 (the issue also appears in Entity Framework 4.2 CTP, but I don’t care about this much now), you can’t use generic providers with Entity Framework. Just to explain, generic providers is pretty much the one way that you can integrate with Entity Framework if you want to write a profiler or a cacher or a… you get the point.

Looking at the code, it is pretty obvious that this is not intentional, but just someone deciding to implement a method without considering the full ramifications. When I found out about the bug, I tried figuring out a way to resolve it, but the only work around would require me to provide a different assembly for each provider that I wanted to support (and there are dozens that we support on EF 4.0 and EF 3.5 right now).

We have currently implemented a work around for SQL Server only, but if you want to use Entity Framework Profiler with Entity Framework 4.1 Update 1 and a database other than SQL Server, we would have to create a special build for your scenario, rather than have you use the generic provider method, which have worked so far.

Remember the beginning of this post? How I said that Backward Compatibility is something that Microsoft is taking so seriously?

Naturally I felt that this (a bug that impacts anyone who extends Entity Framework in such a drastic way) is an important issue to raise with Microsoft. So I contacted the team with my finding, and the response that I got was basically: Use the old version if you want this supported.

What I didn’t get was:

Addressing the issue of a breaking change of this magnitude that isn’t even on a major release, it is an update to a minor release.
Whatever they are even going to fix it, and when this is going to be out.
Whatever the next version (4.2 CTP also have this bug) is going to carry on this issue or not.

I find this unacceptable. The fact that there is a problem with a CTP is annoying, but not that important. The fact that a fully release package has this backward compatibility issue is horrible.

What makes it even worse is the fact that this is an update to a minor version, people excepts this to be a safe upgrade, not something that requires full testing. And anyone who is going to be running Update-Package in VS is going to hit this problem, and Update-Package is something that people do very often. And suddenly, Entity Framework Profiler can no longer work.

Considering the costs that the entire .NET community has to bear in order for Microsoft to preserve backward compatibility, I am deeply disappointed that when I actually need this backward compatibility.

This is from the same guys that refused (for five years!) to fix this bug:

new System.ComponentModel.Int32Converter().IsValid("yellow") == true

Because, and I quote:

We do not have the luxury of making that risky assumption that people will not be affected by the potential change. I know this can be frustrating for you as it is for us. Thanks for your understanding in this matter.

Let me put this to you in perspective, anyone who is using EF Prof is likely to (almost by chance) to get hit by that. When this happens, our only option is to tell them to switch back a version?

That makes us look very bad, regardless of what is the actual reason for that. That means that I am having to undermine my users' trust in my product. "He isn't supporting 4.1, and he might not support 4.2, so we probably can't buy his product, because we want to have the newest version".

This is very upsetting. Not so much about the actual bug, those happen, and I can easily imagine the guy writing the code making assumptions that turn out to be false. Heavens know that I have done the same many times before. I don’t even worry too much about this having escaped the famous Microsoft Testing Cycle.

What (to be frank) pisses me off is that the response that we got from Microsoft for this was that they aren’t going to fix this. That the only choice that I have it to tell people to downgrade if they want to use my product (with all the implications that has for my product).

What is next for the profilers?

Thu, 02 Jun 2011 09:00:00 GMT

We have been working on the profilers (NHibernate Profiler, Entity Framework Profiler, Linq to SQL Profiler, LLBLGen Profiler and Hibernate Profiler) for close to three years now. And we have been running always as 1.x, so we didn’t have a major release (although we have continual releases, we currently have close to 900 drops of the 1.x version).

The question now becomes, what is going to happen in the next version of the profiler?

Production Profiling, the ability to setup your application so that you can connect to your production application and see what is going on right now.
Error Analysis, the ability to provide you with additional insight and common solution to recurring problems.
Global Query Analysis, the ability to take all of your queries, look at their query plans and show your potential issues.

Those are the big ones, we have a few others, and a surprise in store

What would you want to see in the next version of the profiler?