After last night's post about the performance benefits of SqlCommandSet, I decided to give the ADO.Net team some headache, and release the results in a reusable form.

The relevant code can be found here, as part of Rhino Commons. Beside exposing the batching functionality, it is very elegant (if I say so myself) way of exposing functionality that the original author decided to mark private / internal.

I really liked the declaration of this as well:

The usage is very simple:

As a note, I spiked a little test of adding this capability to NHibernate, and it seems to be mostly working, I got 4 (out of 694) test failing because of this. I didn't check performance yet.

I have ranted before about the annoying trend from Microsoft, to weld the hood shut in most of the interesting places. One particulary painful piece is the command batching implementation in .Net 2.0 for SQL Server. The is extremely annoying mainly because the implementation benefits are going for those who are going to be using DataSets (ahem, not me), but are not avialable to anyone outside of Microsoft. (See topic: OR/M, NHibernate, etc).

Today, I have decided to actually check what the performance difference are all about. In order to do this, I opened the (wonderful, amazing) Reflector and started digging. To my surprise, I found that the Batching implementation seems to be centralized around a single class, System.Data.SqlClient.SqlCommandSet (which is internal, of course, to prevent it from being, you know, useful).

Since the class, and all its methods, are internal to System.Data, I had to use Reflection to pry them out into the open. I noticed that the cost of reflection was fairly high, so I converted the test to use delegates, which significantly imporved perfromance. The query I run was a very simple query:

INSERT

INTO [Test].[dbo].[Blogs] ([blog_name]) VALUES (@name)

With the @name = 'foo' as the parameter value. The table is simple Id (identity), Blog_Name (nvarchar(50))

Note: Before each test, I truncated the table, to make sure it is not the additional data that is causing any slowdown.

The Results:

The X axis is the number of inserts made, the Y axis is the number of ticks that the operation took. As you can see, there is quite a performance difference, even for small batch sizes. There is a significant difference between batching and not batching, and that reflection / delegates calls are not a big cost in this scenario.

Here is the cost of a smaller batch:

This shows a significant improvement even for a more real-world loads, even when we use Reflection. 

I just may take advantage of this to implement a BatchingBatcher for NHibernate, it looks like it can make a good benefit for perfromance. Although this will probably not affect SELECT performance, which is usually a bigger issue.

You can get the code here: BatchingPerfTest.txt

I am happy to accounce Rhino Mocks 2.9.1, which came after quite a dry spell...

The changes are:

• Added Message() to Method Options, allowing more structured way to add intent to expectations described in more details here.
• Added operator && and || overloading to constraints
• Added Is.Matching<T>(Predicate<T> pred) constraint
• Better error message for trying to mock a non-virtual method call.
• Fixed stupid issue with BackToRecordAll()
• Removed outdated documentation

I'm not sure if how to explain it, but I actuall go all of this between 04:30 to 05:45 (in the morning).

I also moved all the documentation to the wiki, and at the moment, the greatest help of all would be to add stuff to it.

As usual, the changes can be found here

I got this page when I looked at the logs of an application that was running for about four hours...

Somehow I don't think that the logs will be very useful in this form...

Okay, I think that I managed to dechiper most of the strange behavior in ASP.Net's Http Modules.

• An AppDomain may hold zero or more instances of HttpApplication
• A new instance of HttpApplication may be created (for reasons that I have not been able to figure out) - a set of all the http modules will be created as well.
  
• An instnace of HttpApplication may be disposed (with its associted http modules) at any time.
• A request will always be served by a fully initialized HttpApplication.

The result of this is that you cannot set/unset static variables from an HttpModule or HttpApplication.
Or, more correctly, you can't rely on unsetting a static varaible in Dispose() and setting it in Init() (or Application_Start and Application_End), since they may be called several times.
This mean that you either have to do a ref counting ( Bad ), or implement application level variables in Web secanrios. (Or just forgo about cleaning resources when the HttpApplication is going down).

I chose the first way, but it is not very clean, in my opion, check out the changes that I had to make:

• IoC
• Unit Of work

Basically, this means that when running under Web context, it will use the current application as a global variable, and when running under non-web context, it will use a static variable.

This is what an interface looks like after about two week of work and seventeen refactoring. At one point, it looked like this:

But the fear of the compiler puking on me kept the design a little simpler.

My web.config file defines the following Http Module:

<httpModules>
     <add name="UnitOfWorkModule"
            type="Rhino.Commons.HttpModules.UnitOfWorkModule, Rhino.Commons"/>
</httpModules>

Yet, for some reason, I get two instances of the UnitOfWorkModule, and I can't quite figure out why. It looks like it is proccessed okay, when I look into the Http Application Initialization, I get the following registered modules:


But, if I put a break point on the Init() method in the Http Module, it is called twice! I run it many times, and the "secret" seems to be multiply requests in the application's start. It looks likes several HttpApplication instances are created, and this is what is causing the issue.

This may be related to the previous error, since my state is app-domain global. If the ASP.Net runtime initiate more than a single HttpApplication per app-domain, and clean it up after ward, it may cause the issues that I have seen, of Disposing an HttpModule which hold global state and then not calling the Init() again.

At the moment, I am going to assume that this is the case, which means that my life are ever more complicated than before.
At the moment, I am using a static variable to hold the container instance, but this static variable is shared among all the HttpApplication instances. This probably mean that I need to keep track of it in an Application level variable, but this gets into complicated code in non-web scenarios.

I run into some very strange issues latelt with regard to Http Modules. I am using Http Modules to setup the environment for my application. In particular, I am using Unit Of Work Module to mange my Unit Of Work and the life time of application wide resources (like the Inversion Of Control container instnace).

The problem that I run into was that I began to get errors from the application, saying that the container was missing. That, of course, was not possible, since I registered the HttpModule correctly, and it certainly did it job, for a time.

The issue was mainly that after a while, never determenisticly, the application start throwing errors. But at first, it was smooth sailing. I butt my head against many walls, but I finally manage to get a stack trace of the issue.

Here is the call to Dispose(), which releases the resources required to my application:

The next request after this call, no call to Init() was made, and the application, predictably, crashed.

I googled and search and reflected for quite a while, but I can't figure out the reason for this. I'm pretty certain that if Dispose() is called on the HttpModule, the next request should call Init(), or create a whole new instnace (preferable), but in this circumstance, it looks like Dispose() is called just because it feels like it.

The server is Windows Server 2003, Enterprise, SP1, using ASP.Net 2.0. I have seen this behavior across several machines, including XP one, so I guess it is not the environment that is causing this issue.

At the moment, the only work around is to not use Dispose(), and let resource be cleaned by the Finalizer thread if neccecary.

Any thoughts?

In response for my DDD in Hebrew, I was asked why it is not recommended to code in Hebrew. I decided to question my initial response, and give it a shot. I quickly gave it a try and created the following domain model. I don't expect you to be able to read it (although if you can, you'll find it hilarious).

Here is my DDD in Hebrew, the classes are Employee (right, top), Report (Left, Top), HoursReport (Middle, Bottom), WontBeHereReport (Left, Bottom) and Repository<T> (Right, Bottom):

I run into a couple of issues just with designing this diagram:

• Trying to specify a generic parameter in Hebrew cause issues because of alignments. Basically, the whole class name is presented in the wrong way (this is a common issue with mixing Hebrew & math together).
• Intellisense is mostly useless, check out this, you may not be able to see it, but the Hebrew is half-way reversed.
  
• For some reason Visual Studio is working a lot harder when I write code in Hebrew.

Let us move to more serious issues, take a look at this simple program:

Main issues:

• Hebrew is written right to left, the code here is structured left to right.
• Hebrew doesn't have upper/lower casing, so words need to be seperated by "_".
• I often need to switch keyboard setttings ( even something as simple as "<" has different orientetions in Hebrew and English).
• I need to use primitives (such as DateTime) in English.
• I need to mix English and Hebrew in the same line (see the "new" lines).
• All control flow is in English.
• You can't grep this source. At least not easily. Hebrew and the command line do not mix.

That is even beside the nice point of the enconding hell that you get yourself into. It is very easy to save this as ASCII (if you open it in an editor that doesn't support unicode, or simple use a tool to go over it) and lose all the hebrew characters.

Sorry Eliezer, but you will have to write Hebrew# and language pack, and write a translation layer for the framework if you really want me to use it.

One of the biggest problems with abstractions is that they may allow you to do stupid things without them being obvious. In OR/M-land, that usually means SELECT N+1 issues.
The problem is that you often develop a certain functionality first, and only then realize that while you tested, all was fine and dandy on the five items that you had, but on the real system, you have 5,000, and the DBA is on its way to ER...

Anyway, I am currently working with Web Applications, and I wanted to get a good indication about what pages are troublesome.
Being who I am, I immediately began to design a framework that would correlate page requests to trace data from SQL Server, and then another system that would analyze it and spit out a report saying: "Wow, that ScaryReport.aspx page is making 30% of the calls in the application, take a look at that".

Not wishing to spent the next two years on this project, I decided to do something a bit more modest, and utilize the already existing infrastructure.
In this case ,the infrastructure is NHibernate and lot4net.

The secret tidbit is that NHibernate logs all queries to a log4net logger named "NHibernate.SQL". From there, it was a simple matter of adding a logging helpers to Rhino Commons that would output the current page and the current request id (the request hash code, basically).
Then, It was a matter of defining the following table:

CREATE TABLE [dbo].[NHibernatePerPageQueries](
    [Id] [int] IDENTITY(1,1) PRIMARY KEY NOT NULL,
    [RequestId] [int] NOT NULL,
    [Date] [datetime] NOT NULL,
    [Message] [nvarchar](max)  NOT NULL,
    [PageURL] [nvarchar](max)  NOT NULL
)

Then, to define the appender:

<appender name="NHibernatePerPageAppender"
             type="log4net.Appender.AdoNetAppender">
     <bufferSize value="10" />
     <connectionType value="System.Data.SqlClient.SqlConnection,
System.Data, Version=2.0.0.0, Culture=neutral,
PublicKeyToken=b77a5c561934e089" />
     <connectionString value="Data Source=localhost;Initial
Catalog=Logs;User ID=logger;Password=logger;" />
     <commandText value="INSERT INTO dbo.[NHibernatePerPageQueries]
([Date],[Message],[PageUrl],[RequestId]) VALUES (@log_date, @message,
@pageUrl,@currentRequestId)" />
     <parameter>
           <parameterName value="@log_date" />
           <dbType value="DateTime" />
           <layout type="log4net.Layout.RawTimeStampLayout" />
     </parameter>
     <parameter>
           <parameterName value="@message" />
           <dbType value="String" />
           <size value="4000" />
           <layout type="log4net.Layout.PatternLayout">
                <conversionPattern value="%message" />
           </layout>
     </parameter>
     <parameter>
           <parameterName value="@pageUrl" />
           <dbType value="String" />
           <size value="2000" />
           <layout type="log4net.Layout.PatternLayout">
                <conversionPattern value="%property{nhibernate_page_url}" />
           </layout>
     </parameter>
     <parameter>
           <parameterName value="@currentRequestId" />
           <dbType value="String" />
           <size value="2000" />
           <layout type="log4net.Layout.PatternLayout">
                <conversionPattern value="%property{current_request_id}" />
           </layout>
     </parameter>
</appender>

And defining the logger:

<logger name="NHibernate.SQL">
     <level value="DEBUG" />
     <appender-ref ref="NHibernatePerPageAppender" />
</logger>

We are still not done, though, we need the following in the Application_Start():

GlobalContext.Properties["nhibernate_page_url"] = WebLoggingHelper.CurrentPage;
GlobalContext.Properties["current_request_id"] = WebLoggingHelper.CurrentRequestId;

This is it, now I can correlate the number of queries per page hits, and act accordingly.
Normally, I think that the following queries should be enough:

-- Get pages ordered by number of total queries made from them

SELECT COUNT(*) [Number Of Queries In Page], PageUrl
FROM NHibernatePerPageQueries
WHERE substring(Message,1,1) != '@' -- remove parameters logs
GROUP BY PageUrl ORDER BY COUNT(*)

 

-- Get pages ordered by number of queries per page

SELECT AVG(countOfQueries) [Number Of Queries In Page Per Request], PageUrl FROM (
     SELECT COUNT(*) countOfQueries, PageUrl
     FROM NHibernatePerPageQueries
     WHERE substring(Message,1,1) != '@' -- remove parameters logs
     GROUP BY PageUrl, RequestId ) innerQuery
GROUP BY PageUrl ORDER BY AVG(countOfQueries)

 

Enjoy,