NH Prof
Say hello to Uber Prof
I got several requests for this, so I am making Uber Prof itself available for purchasing. What is Uber Prof? It is a short hand way of saying: All the OR/M profilers that we make. An Uber Prof license gives you the ability to use: NHibernate Profiler Hibernate Profiler Linq to SQL Profiler Entity Framework Profiler And it will automatically give you the ability to use any additional profilers that we will create. And yes, there is an upgrade path if...
What happens behind the scenes: NHibernate, Linq to SQL, Entity Framework scenario analysis
One of the things that I began doing since starting to work on multiple OR/M Profilers is to compare how all of them are handling a particular task. This is by no means a comparative analysis, but it is an interesting data point. The scenario in question is loading a blog with all its posts and comments. Let us start with NHibernate: var blogs = s.CreateQuery(
@"from Blog b
left join fetch b.Posts p
left...
Profiler New Feature: Side by Side diff
The profiler could do session diffs (showing the difference between executed statements between two sessions) for a while now, but we got some requests for changing it to follow a more traditional source control diff style. This is now done, and it should make it easier to understand the changes between two sessions:
UberProf new feature: Query Plan Cache Misuse
This is a new feature available for NHibernate Profiler*, Linq to SQL Profiler and Entity Profiler. Basically, it detects when the same query is executed with different parameter sizes, which generate different query plan in the query cache. Let us say that we issue two queries, to find users by name. (Note that I am using a syntax that will show you the size of the parameters, to demonstrate the problem). We can do this using the following queries. exec sp_executesql
N'SELECT * FROM Users WHERE Username...
ÜberProf new feature: Fully keyboard enabled
For a long time, most of the work in the profiler (NH Prof, HProf, L2S Prof & EF Prof) could be done only with the use of the mouse. That annoyed a bunch of people, but it didn’t really bother me. Rob fixed this recently, and I cannot believe what kind of a difference it makes. Here are the shortcuts: S Focus on Sessions tab header ...
ÜberProf new feature: Programmatic Integration
Well… I am still working done the list of stuff people request for the profiler (NH Prof, HProf, L2S Prof & EF Prof) , and one of the things that popped into the head of the list was wanting to have programmatic access to the profiler output. We aren’t talking about just the XML reports that are available, but to be able to get the data in a way that is easy to work with. Well, here it is: There is a new DLL in the profiler distribution, HibernatingRhinos.Profiler.Integration, where you can find the CaptureProfilerOutput...
NH Prof new feature: Filter static files
One of the most annoying things about doing the TekPub episodes is finding the right session, because NH Prof will detect any session opening. This means that it usually looks like this: Which also means that it is pretty hard to figure out. I got annoyed enough with that to add a specific filter to filter all of those in one shot (you could already do it, but it meant creating several filters): Which results in a much better experience:
ÜberProf and Continuous Integration
One of the features that keep popping up for ÜberProf is that people want to use that in CI scenarios, usually to be able to programmatically check that they don’t have things like SELECT N+1 popping up, etc. With build 562 of ÜberProf, this is now possible. How does this works? We now have a command line interface for ÜberProf, with the following options: /CmdLineMode[+|-] (short form /C)/File:<string> (short form /F)/ReportFormat:{Xml|Html} (short form...
Tweets that warm my heart
I got up this morning and the following showed up in my tweeter search: I would say that from the “is the profiler helping customers?”, we are in a pretty good position.
NH Prof New Feature: Exporting Reports
I mentioned before that exporting reports (so you can mail them around) is something that I would really like to do. In fact, I decided to put it out as a challenge, and I got some really great responses. Josh Perry has been able to flat out amaze me with a solution that was at once simple, elegant and beautiful. It is an HTML report that contains all the styling inline (so it supports emailing it around, for example, to the DBA) but it also has some java script smarts around it that make it behave like the real...
Über Prof goes live
Über Prof is the codename for the multi-OR/M profiler that I have been working on for the last several weeks. Today is a good milestone, because it marks the shift from treating it as an experimental to production ready. Über Prof also brings with it several changes to the way I mange the profiler profiles. Each profile (NHProf, HProf, L2SProf and EFProf, currently) is built on top of the same codebase, but contains slightly different functionality to fits it target audience. I might talk about exactly how we are doing that in a later...
Entity Framework Profiler
I was sitting with Julie Lerman today, and we got into a discussion on how to provide more information to EF users. It appears that there is much need for that. This is what I do, more or less, so we decided to tackle that problem in the bar. Some drinks later, we had a working version of EF Prof that was actually able to intercept all queries coming from the Entity Framework. Initial testing also shows that I’ll be able to provide much more information about EF than I’ll be able to do with Linq to SQL....
Linq to SQL Profiler – Progress
We have mostly done all the infrastructure work that actually needed to be done, what is left is polishing and “i18n” (for lack of a better word). One of the things that I want to do with L2SPRof is to create a different identity from NHProf and HProf. This is me just playing around, not something that is final, but I would still like your thoughts: Two closing notes: It took me about 30 minutes of playing with Blend to get this to work. As usual, Christopher and Rob...
LinqToSql Profiler & NHibernate Profiler – What is happening?
About two weeks ago I posted my spike results of porting NHProf to Linq to SQL, a few moments ago I posted screenshots of the code that is going to private beta.
I spent most of the last week working on the never-ending book, but that is a subject for a different post. Most of the time that I actually spent on the profiler wasn’t spent on integrating with Linq to SQL. To be frank, it took me two hours to do basic integration to Linq to SQL. That is quite impressive, considering that is from the point of view of...
NHProf in the real world: NHibernate Search integration
I had a problem in one of my applications, a Lucene search was returning more results than I wanted it to, and I wasn’t sure what was going on. I pre marked the problematic line, but at the time, I had no idea why it is returning me all those extra results that had nothing to do with my queries. But since NH Prof gave me the full Lucene query text, I could take this and plug this into Luke. Luke is the preferred tool to working with Lucene indexes, and running...
How NH Prof integrate with a profiled application
In order to integrate with an application, NH Prof requires that you’ll reference the appender dll and call NHibernateProfiler.Initialize(). For Linq To SQL, the process is similar, but you call LinqToSqlProfiler.Initialize(). This post describe the internal implementation of how the appender works. Well, that is a lie, I am actually using this post as a design document for the restructuring process that I am currently at to support multiple OR/M backends. The idea is to give some background information so people who want to integrate with the profiler can do so. There are two sides for integrating...
NH Prof New Feature: NHibernate Search Integration
Well, I was demoing how easy it is to add new features to NH Prof in a user group in JAOO today, and tomorrow I am doing my OR/M += 2 talk. Part of the things that I want to show is NHibernate Search, but the demo doesn’t really work unless I can point to NH Prof and show what is going on there. I now consider output results to the console to be so 2005. Here is the code that I am using for this post: ...
Licensing a commercial product
This is another forum question, this time from Brendan Rice: A lot of developers are unsure of how best to go about making money from a product, how do you go about implementing licensing, what pay system do you use, how do you accept payment, are there any legal issues... Well, talk about an open ended question. There are several aspects for the answer, legal, licensing and payment processing. There are somewhat related, though. From the legal side, you need to understand basic concepts in the legal side of software engineering. You need...
Linq to Sql Profiler: Spike results
Well, so far, so good. I started by defining a simple Linq to SQL model, there are zero things that you need to do here to make things work: And now to the actual code using this model: static void Main()
{
LinqToSqlProfiler.Initialize();
using (var db = new BlogModelDataContext())
{
var q = from blog in db.Blogs
where blog.Title ==...
xProfiler – A generic OR/M Profiler
With the release of NH Prof v1.0, I started to look if I can extend what I am doing for NHibernate for other OR/M in the .NET space. My initial spiking makes me optimistic, this is certainly possible. I’ll probably talk at length about the actual architectural implementation, but for now I want to concentrate on the actual high level requirements. I want to be able to support the following: Linq to SQL SubSonic LLBLGen Plug your own DAL While none of them are going...
What is happening with NH Prof?
I was a bit quite on the NH Prof front lately, not because I didn’t work on it, but because I was fighting a really nasty bug. The way NH Prof is making use of WPF exposed a memory leak scenario inside WPF. Luckily, once we were able to isolate the actual problem, it was relatively easy to find a workaround. If you care, the resolution was to keep a single instance bound to the view and replace its values, instead of providing a new instance when replacement was required. The problem is that this bug took...
NH Prof stress testing
Just run NH Prof through a stress test to see how well it handles scenarios where you just throw data at it. I run into a few interesting bugs (mostly related to threading), which were luckily simple enough to handle. The end result was: Which is pretty impressive, I think. Just to give you an idea, we used to test NH Prof against the NHibernate integration test suite, which isn’t even half of what I threw at it now. There is a small problem, however, the profiled application is capable of outputting data...
I knew there were reasons for those multi threading warning lists
One of the warning signs for bad multi threaded code is calling code that you don’t own while holding a lock. I just discovered a bug in NH Prof that is related to just that issue. But I wasn’t really stupid, and the story of this bug is a pretty interesting one. Let us look at what happened. I have a deadlock in the application, where two threads are waiting for one another. In the UI thread, I have: And on a background thread, I have: This is actually code...
NH Prof: The beta mistake
One of the things that I should have done from the get go of the beta program was to force people to upgrade when their version is too old. Why would I want to do that? Because people that continue to use old versions, and old versions has old bugs. Take a look at this error report: This version is almost a hundred builds old, and the error reported was fixed long ago. I also get errors from people using older versions than that. When I built the error reporting feature, I was...
Challenge: NH Prof Exporting Reports
One of the things that I am working on right now is exporting NH Prof reports. This isn’t an earth shattering feature, the idea is to take a report like this: And give you the ability to save it in a format that you can send to your DBA, or maybe just put it away for later perusal. I started with the simplest thing that could possibly work, exporting the reports to XML. Using Linq to XML, that has been a true pleasure, and got me this: ...
Building NH Prof: The background story
I was asked to comment about the background story behind NH Prof, I posted a lot about the technical details about developing NH Prof, but nothing about the actual story and the business decisions that led to developing it. I found myself needing something that would make discovering how NHibernate works almost as soon as I started using NHibernate. That is why I created the NHibernate Query Analyzer. But over time, I realized that what I wanted isn’t actually understanding how NHibernate works, but what it is doing. In particular, I realized that I need to figure out a...
A design answer: What do you do with inaccurate data?
I got a lot of replies for my question about this issue. I intentionally didn’t specify what the problem was, because I wanted to get your input about the basic concept. The problem is related to NH Prof filtering, in particular, filtering is done on the front end of NH Prof, which present some unique problems in one particular area, marked in red in the image: The issue is that statistical data that is calculated on the backend in a very efficient manner is now potentially invalid. Now, we could just remove...
NH Prof: Post v1.0 Ideas
This is just a high level list of features that I want to plug into NH Prof, now that the v1.0 mark is behind us. Show database query plan for statements Allow occasionally connected profiling, also known as production profiling – note, must be secured(!) Output reports as PDF/HTML Exporting reports NHibernate Search integration NHibernate Shards integration This isn’t everything, but it would serve as a good place for discussion. Licensing note: this would...
Automation, Mark II
It just hit me that I just finish executing the entire NH Prof integration test suite on my new machine, and I had done absolutely no setup. Well, I tell a lie. I had to configure the connection string, like this: It isn’t very smart approach, but it is the one that has the least friction. Once I did that, all I had to do next was to execute the integration test suite. It is smart enough to recognize that it is running on a new machine (originally created because I wanted to save the other members...
NHibernate Profiler v1.0 Released
Well, it is about time, isn’t it? The NHibernate Profiler started about a year ago, and I estimated it at about three months of development. We released the public beta version in that time frame, and I think that we could have called in v1.0 then. In the last six months we worked on adding new features (filters, alerts, detecting caching, etc), significantly improved performance and in general worked on polishing the product. There are always more things to do, but at some point you have to cut the umbilical. NH Prof contains everything...
How many bugs can a single assumption cause?
NH Prof make some assumptions about the way you use NHibernate. In particular, it assumes that you are using it in a best practice way and will warn you if you deviate from that. I just got a bug report in which NH Prof was reporting a cross threaded session usage when no such thing actually happened. Once I got the dump, I took one look at the actual data and immediately know what was going on there. It looked like this: And I went Duh! We have a distributed transaction here, and the...
NH Prof Beta ends in two days
This is just a reminder, but in two days (Friday), the NH Prof beta will end, and the 30% discount for the beta will go away. If you want to buy NH Prof, you probably want to buy it before Friday. Although, if you want to wait a week, I am fine with that too :-)
Commercial Support for NHibernate
One of the oft repeated objections to using NHibernate is that there isn’t someone that you can officially complain to. Some organizations put great store in having someone to officially complain to :-). The NHibernate community, especially the NHibernate Users mailing list is active and thriving, and a large percentage of the support questions that you may have can be answered there, for free. However, there are situations where you really do want to have someone to call to get you out of the foxhole. And there are organizations that will not accept a technology unless they...
NH Prof: Feature Freeze
NH Prof just went for a feature freeze in preparation for a release. This means that sometimes next week, the v1.0 release will go up. To be rather more exact, the v1.0 is out as far as the bits are concerned. Build 414 is going to be it, unless people find some big bugs in it. There are two main reasons for the delay. No one likes to have to push out a v1.0.1 release, so it is better to play it safe. For the beta period, a full license...
NH Prof: The ten minutes feature
15:51 - I have about ten more minutes before starting a presentation, and I thought that I might as well make use of the time and do some work on NH Prof. This feature is supporting filtering of sessions by URL. And I don’t expect it to be very hard. 16:01 - Manual testing is successful, writing a test for it 16:02 – Test passed, ready to commit, but don’t have network connection to do so. The new feature is integrated into the application, in the UI, filtering appropriately, the works: Just...
Concepts & Features: A concept cover the whole range, a feature is constrained
A while ago I mentioned the idea of Concepts and Features, I expounded it a bit more in the Feature by Feature post. Concepts and Features is the logical result of applying the Open Closed and Single Responsibility Principles. It boils down to a single requirement: A feature creation may not involve any design activity. Please read the original post for details about how this is actually handled. And a the specific example about filtering with NH Prof. But while I put constraints on what a feature is, I haven’t talked about what...
Concepts & Features in NH Prof: Filtering
A while ago I mentioned the idea of Concepts and Features, I expounded it a bit more in the Feature by Feature post. Concepts and Features is the logical result of applying the Open Closed and Single Responsibility Principles. It boils down to a single requirement: A feature creation may not involve any design activity. Please read the original post for details about how this is actually handled. In this case, I wanted to show you how this works in practice with NH Prof. Filtering in NH Prof is a new concept, which allows...
NH Prof, Error Handling, Continuous Integration and RTM per commit
One of the nice things about NH Prof is the way that it is setup from an infrastructure stand point. Take a look at the following email: I got this email, along with an additional one containing the actual crash information (this email is generated from the user’s input about the reason for NH Prof crashing). This isn’t the first time that I get this bug (it is an annoying threading issue) but this time I was able to finally pinpoint the exact place where this is happening, trace it back to where it is...
NH Prof New Feature: Multiple Session Factory Support
One common request for NH Prof is that we will support multiple session factories. This turn out to be a bit more problematic than it would seems, mostly because NHibernate doesn’t currently output this relation in any way that NH Prof can access. But here is the first step along the way, NH Prof can now report statistics from multiple session factories.
NH Prof Feature Selection
We are considering the User Interface for the session diff feature for NH Prof, we have two possible user interface, and I would like to solicit some feedback about which of them you like more. I started with this one, called the tabbed interface: And we also have the diff view interface: One thing that we will not have is an option to select between the two, I consider “just make it an option” a mark of ineffective project leadership. Thoughts? Update: People asks about having a filter option, here...
NH Prof New Feature: Diffing Sessions
It is very common to use NH Prof as.. well, a profiler :-) That is, we execute a piece of code, look at the profiler output, and try to improve it. The problem is that when you are working on big applications, a single request may perform quite a bit of work. That can make it harder to figure out the changes between the first session and the optimized session. In order to make this easier, I added a feature that allows you to compare two sessions, see what statements were added, what were removed and what...
NH Prof New Feature: Capturing DDL
This came up as a request yesterday, and I spent a few minutes making it happen today. NH Prof can now capture the output of Schema Export and Schema Update:
NH Prof New Feature: Detect Cross Thread Session Usage
NHibernate sessions are not thread safe, and attempting to use them in multiple threads requires careful synchronization. It is generally better to consider a session only useful within the thread that created it. There are valid scenarios for cross thread session usage (background loading with careful sync, multi request spanning session), but the invalid scenarios are far more common. Let us take a look at the following code: public class CrossThreadSession : IScenario
{
public void Execute(ISessionFactory factory)
{
using(var session = factory.OpenSession())
{
var @event = new ManualResetEvent(false);
ThreadPool.QueueUserWorkItem(state =>
{
using(var tx = session.BeginTransaction())
{
session.Get<Blog>(1);
tx.Commit();
}
@event.Set();
});
@event.WaitOne();
}
}
}
If you’ll try to execute it with NH...
NH Prof: Query Duration
One of the more popular requests for NH Prof was to be able to track query duration not only for select statements, but for all statements. It took a bit of work, I’ll admit, far more than you would probably assume from the screen shot, but it is here: What you can see now is that the duration is divided into the time a query took in the database and the total time a query took. For the most part, there wouldn’t be much of a difference between them, but it is occasionally interesting...
NH Prof: Exposing the data that you already have
One of the things that drove me mad with NH Prof is that at times it was hard to visually see when a session went to the database a lot or hit the cache frequently. I mean, you most certainly had that information if you drill into the session itself, but there is really no excuse for not giving that information to the user directly. Hence, the following screen shot: I am back to working on NH Prof full time, and I love the way everything is set up. Time to feature is ridiculously low.
NH Prof: Getting big, and bigger
Well, at least now I have proof positive that all the architecture work that we did in order to ensure that NH Prof will scale better actually works. The following screenshot shows an example that used to crash NH Prof, it loading an 80 MB file with over 200,000 individual events to scan and process. The fun part is that this is run through the entire pipeline, but without involving the UI, so we get the ability to process all those events without having to wait for the UI. If you remember, we had to...
NH Prof: NHibernate Profiler vs. Sql Profiler
I recently got a question, asking why someone should pay for NH Prof if SQL profiler is available for (effectively) free? The answer is actually very easy to answer. I run the following scenario through both NH Prof and SQL Profiler: using (var s = sf.OpenSession())
using (var tx = s.BeginTransaction())
{
s.CreateCriteria<User>()
.Add(Restrictions.NaturalId()
.Set("Username", "Ayende"))
.SetCacheable(true)
.UniqueResult();
tx.Commit();
}
using (var s = sf.OpenSession())
using (var tx = s.BeginTransaction())
{
s.CreateCriteria<User>()
.Add(Restrictions.NaturalId()
.Set("Username", "Ayende"))
.SetCacheable(true)
.UniqueResult();
tx.Commit();
}
Here is what NH Prof gave me:
And here is what I got from SQL Profiler:
I think that I have rested my case, pretty conclusively.
NH Prof: Architectural Overview
I am pretty fond of the way NH Prof is architected. At a very high level, it looks like this: The first part isn’t really that interesting, the main challenge is to get the data from your application to the profiler in as speedy and performant way as possible. We are using a binary protocol based on Protocol Buffers with TCP Sockets as the communication mechanism. Next we have the listener and the message processors. That is where a lot of the magic actually happens, we do first level analysis on the...
NH Prof & Teaching NHibernate
I have taught NHibernate both before and after NH Prof was available. I have to say, there is absolutely no way that I can compare the two. NH Prof, I know that I am not suppose to say that, but I love you. I really love you! I am saying this after spending 4 days doing intensive NHibernate stuff, a full 3 days course and NHibernate consulting day. And NH Prof made it so much easier that I cannot really describe. Participants in my course can testify how at several points I...
NH Prof: Why you should use it
This is a screen shot one of NH Prof’s users has sent me:
A picture is worth a thousand words, I believe is the saying. But in this case, I think it is worth about 1755 queries.
Can you make money on OSS tooling in the .NET world?
The answer is yes. Here is a chart of downloads and orders of NH Prof. I am fairly happy with the way NH Prof sells. I think it could be better, but I want to see what happens to sales when I release the v1.0 version (which will be soon). Nitpicker corner: Numbers has been removed from the chart for a reason.
NH Prof New Feature: Detecting 2nd cache collection loads
I noticed the lack when I was teaching the Core NHibernate course. While NH Prof was able to detect loading of an entity from the second level cache and the loading of query from the query cache, it wouldn’t show when we were loading a collection from the collection cache. That was actually quite easy to fix :-) Take a look: This feature is now available on Build 277 and up. Have fun, and happy cache tracing :-)
NH Prof: Release Candidate is out
Well, After quite a while of developing on a private branch, it is time to make NH Prof RC public. We have worked on several new features, but chief among them was performance. As I outlined in the past, we have to rip apart the internal communication mechanisms inside NH Prof, to allow it to react well under high load. There are some things that you need to know first, though: Session factory stats is currently disabled, there is no reason except that some things had to be cut in order to release...
NH Prof for Hibernate – Now in Closed Beta
Well, I heard the requirement quite clearly, so let us see how this looks like, given the following code: public class SelectBlogByIdUsingCriteria extends AbstractScenario {
@Override
protected void doExecute(SessionFactory factory) {
doInTransaction(new SessionCallback<Object>() {
@Override
public Object execute(Session session) {
return...
NH Prof New Feature: Error Detection
The feature for the day is detecting & notifying the user about NHibernate errors, warning messages and other things that require human attention. As a simple example, let us take this common error: public class ErrorSavingToDatabase : IScenario
{
public void Execute(ISessionFactory factory)
{
using(var s = factory.OpenSession())
using(var tx = s.BeginTransaction())
{
s.Save(new...
NH Prof New Feature: Queries by Url
I have been talking a lot lately about the technical aspects of working with NH Prof (and there are more posts scheduled), but not really talking about the new features at all. I firmly believe in the lynched-by-the-users model, and working for so long without having new features is an anathema to me. I am going to try to leak them to the blog as they go fully online (with screenshotable UI), although I do mean to keep several surprises in my sleeve, and in my backpack, too, come to think about it, I need space for some...
NH Prof: Persistence Format
As someone that so firmly believe that persistence is a solved problem, I keep tripping over it. The issue is quite simple, each scenario has radically different requirements, and usually require different solutions. In NH Prof case, just using RDBMS is not a really good solution, but before arriving to that conclusion, I really need to explain what the requirements are. For NH Prof, there are several reasons to want to be able to persist things: Create an offline dump of the profiling session, to be analyzed later. ...
NH Prof feedback
Every now and then I am running a search to see what people are thinking about NH Prof. And I decided that this time it might be a good idea to share the results found on Twitter. I am still looking for more testimonials, by the way. You are welcome :-) I am not sure if that is that much of a good thing, though… I certainly agree :-) Should I worry about it or use it as a marketing channel? Remind me something from...
NH Prof: An important milestone
Yeah, this now works: And if you wonder why I am happy about something that looks very similar to what already worked months ago, you are right. What you don’t see is that this version of NH Prof uses the revised backend, which I have talked about before. This backend use the pull vs. push mechanism, and it is supposed to allow us much higher performance than before. And no, this branch is not merge to the public builds yet. Give us a few weeks.
Application structure: Concepts & Features
I spoke about this topic quite a bit in ALT.Net Seattle. This is mostly relating to application architecture and how you structure your application. This is the logical result of applying the Open Closed and Single Responsibility Principles. Feature may not be as overloaded a term in our industry as a service, but it still important to define exactly what I mean. A feature, in my eyes, is a discrete part of the application that perform some operation. That is pretty vague, I know, but I hope that the examples I am going to go through later would make...
Closed source?
Some interesting observations about my experience in shipping commercial software. It seems like the notion of closed source isn’t really working. Torkel has developed an application using the NH Prof architecture, peeked via reflector. And no, I don’t really mind. The part that really blew me away was when I started getting patches for NH Prof. No, they didn’t have the code, but they were able to: Follow a pretty complex internal interaction Isolate the actual part that caused their problem Provide a solution and...
Presenting NH Prof at Virtual ALT.Net
On Wednesday 18 Feb 21:00 (GMT-6) I will be joining the weekly VAN.
I am going to show off NH Prof in the context of a real application, discuss its features and how you can use it.
NH Prof: Performance implications for the profiled application
I got a couple of complaints about this, and I think that this is a really important issue. Using NH Prof slow down the profiled application. That is to be expected, since we require the application to do more work, but it is not as good as we can hope it would be. Here is the test scenario, running against the Northwind database (thanks Davy for supplying it): class Program
{
static void Main(string[] args)
{
var sp = Stopwatch.StartNew();
HibernatingRhinos.NHibernate.Profiler.Appender.NHibernateProfiler.Initialize();
ISessionFactory sessionFactory = CreateSessionFactory();
using (var session = sessionFactory.OpenSession())
{
var products = session.CreateCriteria(typeof(Product))
.SetCacheable(true).List<Product>();
}
using (var session = sessionFactory.OpenSession())
{
var products = session.CreateCriteria(typeof(Product))
.SetCacheable(true).List<Product>();
}
using (var session = sessionFactory.OpenSession())
{
var products =...
NH Prof Stats
I am just doing some work on NH Prof, and I run into the line counter button, and I thought it is interesting. Here is the break down of line counts in the application: Appender - 818 lines of codeThe Appender is responsible for communication with the profiled application Backend - 3,548 lines of codeThe Backend is responsible for translating the event stream from the Appender into a coherent idea, apply analysis and send them to the UI. Client - 6,430 lines of codeThe Client is where all the UI work is done, it contains the view model, which...
Tracking NH Prof performance problem on aggressively active application
I mentioned before that NH Prof has a problem with dealing with a lot of activity from the profiled application. In essence, until it catch up with what the application is doing, there isn't much that you can do with the profiler. I consider this a bug, and I decided to track that issue with some verifiable results, and not just rely on hunches. The results were... interesting. The actual problem was in the place that I expected it to be, but it was more complex than expected. Let me try to explain. The general architecture of NH Prof is...
Challenges when building NH Prof
Right now, we have one challenge with building NH Prof: making the UI scale to what people are actually using. The problem that we face is: A profiled application is generating a lot of events. Many of those events are relevant to the profiler and so result in UI Model changes. UI Model changes require UI refresh. There is a limit on how often you can refresh the UI. It is not just a matter of the UI being incredibly slow. (Try it, UI is about as slow as remote system calls!) That problem we already solved with batching...
Better to release early and be ridiculed than just ridiculed
The title for this post is taken from this post. I released NH Prof to the wild in two stages. First, I had a closed beta, with people that I personally know and trust. After resolving all the issues for the closed beta group, we went into a public beta. Something that may not be obvious from the NH Prof site is that when you download NH Prof from the site, you are actually downloading the latest build. The actual download site is here. NH Prof has a CI process that push it to the public whenever anyone makes a...
NHProf, Open Source, Licensing and a WTF in a good sense
I have a Google alert setup for NH Prof. I got the following alert. I was willing to mutter a few choice curses and let it go, because there really isn't much that you can do about this. But then I followed up on the rest of the thread. Um... thanks? I mean, I sure appreciate the sentiment. But the fun continues... And the replies... Honestly, I wouldn't believe it if I didn't see it with my own eyes. acexman & cluka, thanks. Oufti, I don't think that I like you very...
A Customer Service Story
Continuing the XHEO saga, I finally replaced the licensing component for NH Prof. Yes, there is Rhino Licensing, I am sadden to say. Although an open source licensing component seems to be... an interesting contradiction. It would be a great joke to make it GPL as well, and see what happens. Anyway, this is not the point of this post. Currently, the change is committed to the repository and I already updated the NHProf.com backend to generate the new licenses instead of the old ones. That is where we get into a problem. We don't have UI for the new...
NH Prof New Feature: View Query Results
At last, we have a cool UI for this feature. It starts here: And then moves to this window: Yes, I know, it is not earth shattering UI, for sure, but it is one hell of a cool way to inspect even deeper into what NHibernate is doing.
NH Prof New Features: Disabling & Ignoring Alerts
Those are not actually new features, if you want to be strict about it. There is a whole bunch of things in NH Prof that already exists, but are only now starting to have an exposed UI. I believe that NH Prof’s ability to analyze and detect problems in your NHibernate’s usage is one of the most valuable features that it have. Heavens know that I spent enough time on that thing to make it understand everything that I learned about NHibernate in 5 years of usage (how did it get to be that long)? The problem...
NH Prof new feature: Superfluous <many-to-one> update
Yes, I am aware that I said that I would only have two more feature for NH Prof before releasing. But I am currently being held hostage by the new features fairy, and negotiations over a feature freeze seems to have gotten to a stand still. Beside, it is a neat feature. The actual feature is quite simple. Let us say that we have the following model: Notice that this is a very common case of bidirectional association, and this is mapped to the following table model: Notice that while on the...
It ain’t so simple, mister!
The most requested feature for NH Prof is the ability to view the query results inline. I created a mockup just to show how this might look like. The real discussion is below the image. On the face of it, this feature looks very simple. Query the database, throw the results into a grid, done. It is actually not that simple. Let me enumerate all the things that complicate this feature. The simplest thing is a big stumbling block. How do I get the connection string? I can make the profiler figure out...
NH Prof: Balancing functionality, simplicity and form
There are two features that I want to get done before I call a feature freeze on NH Prof and just deal with any bugs and improvements that come up until I feel it is mature enough to call it v1.0. One of them is filtering capability. This was a pretty common request once people started realizing the kind of things that they can do with NH Prof. Ad hoc filtering into NHibernate’s activity can bring up a lot of insight, and I certainly think that this would be an good feature to have. The...
NH Prof new feature: URL tracking
Now that is what I call a hard to build feature. Well, it wasn’t hard, it was just tedious to do. This feature required me to modify 66 files(!). Since I pride myself on the mostly frictionless nature of NH Prof, that was annoying. The real issue was that this required that I would change all layers in the application to start tracking the URL from which the event was generated. Since we have a rigid separation between the different parts, and since we track so many things, it was mainly an annoying task to go and add a URL...
NH Prof documentation: Transaction and the second level cache
Another implication of not using explicit transactions with NHibernate is related ot the use of the second level cache. NHibernate goes to great lengths in order to ensure that the 2nd level cache shows a consistent view of the database. In order to do that it is defering all 2nd level cache updates to the transaction commit. In that way, we can be sure that the data in the 2nd level cache is the one committed to the database. Forgoing the use of explicit transactions has the effect of nulling the 2nd level cache. Here is an example...
NH Prof New Feature: Detecting distributed transactions (System.Transactions)
Here is another nice one: And the disclaimer is that this works on NHibernate from r3981 or higher. I did some work on NHibernate's System.Transactions support recently, and that was one of the side results.
NH Prof New Feature: The Query Cache
Okay, I said it is not there yet, but I got annoyed by the lack of this feature, and I really had to make this work. That one was a pretty tough one. It required me to make some minor modifications to NHiberante, as a result, this feature works with NHbierante r3976 or up only. What this means is that if you want to use this feature, you have to get it from the trunk. This is the only feature in NH Prof that requires the trunk. In this feature, we can see that we detect a cached query, and...
NH Prof feature: The 2nd level cache
One word of warning first, this feature is currently for entities only, NH Prof cannot (currently) inspect cached queries. That said, I am working on it, but this is something that is going to require some changes to NHibernate, so it is a feature that will be available on NHibernate 2.1 only. That said, you can see the profiling result of this piece of code: public class LoadEntityByIdFromSecondLevelCache : IScenario
{
public void Execute(ISessionFactory factory)
{
int id;
using (ISession s = factory.OpenSession())
using(ITransaction tx = s.BeginTransaction())
{
id = s.CreateCriteria(typeof(Blog))
.SetMaxResults(1)
.UniqueResult<Blog>().Id;
tx.Commit();
}
using (ISession s = factory.OpenSession())
{
var blog = s.Load<Blog>(id);
Console.WriteLine(blog.Title);
}
}
}
And, the actual profiler output:
We can clearly see that we...
NH Prof is version tolerant
From the get go, I decided that I wanted NH Prof to work without having an explicit dependency on a particular version of NHibernate. I had trouble with that when I built the NHibernate Query Analyzer, and I had enough of that. This is especially true since I am one of the target audience of NH Prof, and I would hate having to manage NH Prof <---> NHibernate version mapping. So I set out to make NH Prof version tolerant. My explicit goal was to be able to work with NHibernate 2.0 and up. Along the way, I did something...
NH Prof New Feature: Query Duration
I keep adding new features and fixing strange bugs in NH Prof. Most of those aren't really interesting things.
Does anyone really care about sort order in a particular query? Those are things that are important for the overall UX, but they are not things that I can honestly call them features.
This thing, however, is a full blown feature. Even if it takes just a tiny bit of screen real estate.
And integrating it into the analysis section of NH Prof as well:
And in the views by method report:
And something that is important to remember, the moment that...
NH Prof is in public beta
Surprisingly, the hardest part in getting here wasn't writing the actual software, but all the other activities around it. Setting a web site, integrating a payment provider, getting content for said web site, etc. But, we are now there, and NH Prof is in public beta, hooray! You can go to the NHibernate Profiler web site and download a 30 days trail right now. A few important points: It is still in beta. There are several issues that we are aware of in the current version that will be fixed before we ship v1.0. The reason it...
NH Prof Alerts: Use statement batching
This is a bit from the docs for NH Prof, which I am sharing in order to get some peer review. This warning is raised when the profiler detects that you are writing a lot of data to the database. Similar to the warning about too many calls to the database, the main issue here is the number of remote calls and the time they take. We can batch together several queries using NHibernate's support for Multi Query and Multi Criteria, but a relatively unknown feature for NHibernate is the ability to batch a set of write statements into...
NH Prof new features: Overall usage and aggregated alerts
Hat tip to Chad Myers for requesting those features. Here is an example of the overall usage report for the entire application. The aggregated alerts gives you a view about how your application is doing. As you can probably see, this isn't a very healthy application. But since this is actually showing the test suite for the profiler, I am happy with that. On the bottom, you can also see what entities where loaded throughout the entire profiling session, although this is not their final look & feel. We can also see inspect the details of a...
NH Prof Alerts: Too many database calls per session
This is a bit from the docs for NH Prof, which I am sharing in order to get some peer review.
One of the most expensive operations that we can do in our applications is to make a remote call. Going beyond our own process is an extremely expensive operation. Going beyond the local machine is more expensive yet again.
Calling the database, whatever to query or to write, is a remote call and we want to reduce the number of remote calls as much as possible. This warning is being raised when the profiler notice that a single session is...
NH Prof Alerts: Excessive number of rows returned
This is a bit from the docs for NH Prof, which I am sharing in order to get some peer review. The excessive number of rows returned is a warning that is being generated from the profiler when... a query is returning a large number of rows. The simplest scenario is that we simply loaded all the rows in a large table, using something like this code: session.CreateCriteria(typeof(Order))
.List<User>();
This is a common mistake when you are binding to a UI component (such as a grid) that perform its own paging. This is a problem is several levels:
We tend to want...
NH Prof Alerts: Unbounded result set
This is a bit from the docs for NH Prof, which I am sharing in order to get some peer review. Unbounded result set is perform a query without explicitly limiting the number of returned result (using SetMaxResults() with NHibernate, or using TOP or LIMIT clauses in the SQL). Usually, this means that the application is assuming that a query will only return a few records. That works well in development and testing, but it is a time bomb in production. The query suddenly starts returning thousands upon thousands of rows and in some cases, it is...
NH Prof Alerts: Use of implicit transactions is discouraged
This is a bit from the docs for NH Prof, which I am sharing in order to get some peer review. A common mistake when using a database is that we should use only transactions to orchestrate several write statements. Every operation that the database is doing is done inside a transaction. This include both queries and writes ( update, insert, delete ). When we don't define our own transactions, we fall back into implicit transaction mode, in which every statement to the database run in its own transaction, resulting in a higher performance cost (database time to build...
NH Prof Alerts: Select N + 1
This is a bit from the docs for NH Prof, which I am sharing in order to get some peer review. Select N+1 is a data access anti pattern, in which we are accessing the database in one of the least optimal ways. Let us take a look at a code sample, and then discuss what is going on. I want to show the user all the comments from all the posts, so they can delete all the nasty comments. The naןve implementation would be something like: // SELECT * FROM Posts
foreach (Post post in session.CreateQuery("from Post").List())
{
...
NH Prof Deep Dive: Implemented the Unbounded Result Set warning
So, I talked a bit about the architecture and the actual feature, but let us see how I have actually build & implemented this feature. This is the actual code that goes into the actual product, I want to point out. And this is actually one of the more complex ones, because of the possible state changes. public class UnboundedResultSetStatementProcessor : IStatementProcessor
{
public void BeforeAttachingToSession(SessionInformation sessionInformation,
FormattedStatement statement)
{
}
public void AfterAttachingToSession(SessionInformation sessionInformation,
FormattedStatement statement, OnNewAction newAction)
{
if(statement.CountOfRows!=null)
{
CheckStatementForUnboundedResultSet(statement, newAction);
return;
}
bool addedAction = false;
statement.ValuesRefreshed += () =>
{
if(addedAction)
return;
addedAction = CheckStatementForUnboundedResultSet(statement, newAction);
};
}
public bool CheckStatementForUnboundedResultSet(FormattedStatement statement,
OnNewAction newAction)
{
if (statement.CountOfRows == null)
return false;
// we are discounting statements returning 1...
NH Prof Deep Dive: Applying the Open Close Principle at the architecture level
The back end in NH Prof is responsible for intercepting NHibernate's events, making sense of all the mess, applying best practices suggestions and forwarding to the front end for display. Second, it is also a good example of how I apply the Open Close Principle at the architecture level. With NH Prof, there are multiple extension points that I can use to add new features. Here is a schematic of how things works: Not shown here is the NHibernate Listener (of which, of course, I have several), which is publishing events to the bus. Those events are first...
NH Prof new feature: Unbounded result sets
One of the things that I am doing with NH Prof is not only giving you visibility into what NHibernate is doing, but also trying to automate my own experience in analyzing best practices and problematic usages of NHibernate. NH Prof will detect usage patterns and warn against bad practices and suggest how to deal with them. The first one that I implemented was detecting Select N+1, and the feedback from the beta group was "Wow! I didn't even know that we had this problem, but casual use with the profiler immediately showed it." Here is the newest feature, detecting...
NH Prof new feature: Row Counts
That was actually a hard to implement feature, since this is not something that NHibernate just give out. Nevertheless, by trawling through the codebase long enough, I was able to figure out how to do this. As an aside, one of the requested features for NH Prof was to be able to get DB level stats. I am not going to do this for v1.0, but I now have a pretty firm idea about how to implement this. We will have to see how many people request this information.
NH Prof New Features: Analysis
I have been quite on that front for a while, but not because we stopped working on that, but because we were quietly adding features and polishing what we have to reach a whole new level. Some of those features are not yet complete, but we have one feature that is good enough to show. Analysis In this case, a picture is worth significantly more than a thousand words. Here is one of the analysis modes that we support, showing you all the unique queries that you application perform. But wait, there is more. The (tentatively...
NH Prof Public Beta Date
Following up on my previous post about shipping being the major success criteria for software, I wanted to announce the public beta for the NHibernate Profiler. And the date is... by the end of the year. Which gives me about a week to get everything together & presentable. I am following Parkinson's law which states that: Work expands so as to fill the time available for its completion. You are going to see some new things as well about it soon.
NH Prof: Getting to zero friction
Here is a new (passing) integration test in the NHibernate Profiler:
Just to give you an idea, here is the implementation of this rule, which has to be one of the more complex ones that we have at the moment, because the data for it comes from several sources, so we need to actually execute the logic in an event, instead of directly.
public class TooManyRowsReturnedPerQuery : IStatementProcessor{ private ProfilerConfiguration configuration; public TooManyRowsReturnedPerQuery(ProfilerConfiguration configuration) { this.configuration = configuration; } ...
NH Prof: A moment in time
I just had a major moment in using NH Prof.
I run into a problem with NHibernate and was able to use the NH Profiler in order to figure out what the problem was.
Wow!
NHProf: What is the role of the DBA?
I am now discussing what sort of reports we want to give the DBA from the NHibernate Profiler. My first thought was just to give the DBA a list of the statements that the application has executed and the number of times they were repeated. That should allow him to get enough information to use his own tools to optimize the application physical data structure.
What do you think? Is this a good scenario?
What other scenarios can you see for NH Prof in the hands of the DBA?
NHProf: The stack is not as simple as you wish it to be
One of the nicest features that NH Prof has to offer is this, allowing you to go from the query issued to the database directly to the line of code that caused this query to be generated.
A few days ago I posted that you can either build something in one day, or in three months, but nothing much in the middle. The proof of concept that convinced me that I can build NH Prof was written during a single evening, along with two pints of Guinness. The overall concept that I have now is drastically different, but it is...
NH Prof: Configuration Story
I think that I mentioned that NHibernate Profiler is working mostly by doing some smarts on top of the log output from NHibernate. That is not exactly the case, but that is close enough. The problem with working through the logs is that there are roughly 30 lines of XML that you need to deal with in order to manage this properly.
The first time I sent this to anyone else, he run into problems with the configuration because of very subtle issues. For a while now, I had a ticket saying that I need to document what the failure...
NH Prof: A testing story
Remember that I mentioned the difference about working and production quality?
One of the things that separate the two that in production quality software, you don't need to know which buttons not to push. Here is a simple example. For a while now, if you tried to bring up two instances of NH Prof, the second one would crash. That wasn't something that you really want to show the users. Today I got back to doing NH Prof stuff, getting it ready for public beta, and I decided that the first thing to do was to tackle this easy feature.
Doing...
NH Prof Deep Dive: The Integration Test Architecture
I am getting a lot of requests to explore the actual innards of the NH Prof. I find it surprising, because I didn't think that people would actually be interested in that aspect of the tool.
But since interest was expressed, I'll do my best to satisfy the curiosity.The first topic to discuss is the integration test architecture. One of the things that the profiler is doing is to capture data from a remote process, and I wanted my integration tests to be able to test that scenario, which exposes me to things like synchronization issues, cross process communication and (not...
An NH Prof Bug Story: Why integration is tricky
Today I found out that NH Prof doesn't work with ASP.Net MVC applications. If you would have asked me, I would have sworn any oath you care to name that it would. And even after seeing the problem with my own eyes, it too me a while to track it down.
To make a long story short. Somewhere deep in the bowels of NH Prof, I made the assumption that a method is always contained in a type. I am pretty familiar with the way that the CLR works, and it seemed like a pretty reasonable assumption to make. In fact,...
NH Prof: A guided tour
NH Prof has reached the level in which I can actually talk about the features that it has in more than abstract terms. There is still a big feature area that I want to cover (which should be a nice surprise), but the basics are there, and today I had ample proof that it is maturing just nicely. I was able to deal with quite a few of the remaining tasks by applying check listing. Basically, to do X, I had to do A,B & C. Trivially simple, and quite satisfying.
Test coverage went back up to over 90% on...
NH Prof: Teaser
If you want to learn more, come to my Advanced NHibernate talk tomorrow.
This time, this is literally a snapshot of the application as it is running, and it is showing most of the surface level functionality that exists at the moment in the application.
Oh, and all the kudos for the look and feel goes to Christopher and Rob, who make it looks so easy.
NH Prof: How to detect SELECT N + 1
One of the things that the NHibernate Profiler is going to do is to inspect your NHibernate usage and suggest improvements to them.
Since I consider this to be a pretty important capability, I wanted to stream line the process as much as possible.
Here is how I detect this now:
It is not perfect, but it is pretty close.
A messaging problem: Order in the bus
In NH Prof, I have structured the application around the idea of message passing. I am not yet in the erlang world (which requires a framework that would keep hold of the state), but I am nearer than before.
The back end of the profiler is listening to the event streams generated by NHibernate. We get things like:
Session opened on thread #1
SQL Executed on thread #1
SELECT * FROM Customers
...
NHProf: Logging interception
One of the goals that I set for myself with the NHibernate Profiler is to be able to run on unmodified NHibernate 2.0. The way that I do that is by intercepting and parsing the log stream from NHibernate.
NHibernate logging is extremely rich and detailed, so anything I wanted to do so far was possible. I am pretty sure that there would come a time when a feature would require more invasive approaches, running profiler code in the client application to gather more information, but for now this is enough.
I did run into several problems with logging interception. Ideally,...
NHProf: Another milestone
It is by no means the final UI, it was just thrown together by me in about half an hour, just to show how things are working.
I am also using it now to track what is going on in an application that I am working on.
NHProf: Alive! It is alive!
I just finished writing the final test for the basic functionality that I want for NHibernate Profiler:
[Test] public void SelectBlogById() { ExecuteScenarioInDifferentProcess<SelectBlogByIdUsingCriteria>(); StatementModel selectBlogById = observer.Model.Sessions.First() .Statements.First(); const string expected...
The NHibernate Profiler
This is speculative at the moment, just to be clear.
I am thinking about creating a profiler for NHibernate. This came out of the common need to actually get a good view about what is going on with NHibernate.
This is intended to be a commercial project.
I have a feature set in mind, but I would rather hear from you if you think that it is a tool that you would use (and buy) and what kind of features do you expect such a tool to have?
And, to forestall the nitpickers, I am well aware of SQL Profiler.