Ayende @ Rahien

It's a girl

NHProf.com downtime, and using windows as a server system

Yesterday nhprof.com was down, there was a YSOD and when I logged into the server to try to figure out what the problem was, I got this:


The server has been running with no issues for over a year, so it is not an issue with not activating after installation. This is a Windows 2008 Server, and I find it incovievable that a server would go down because of activation. To spread more salt on the wound, it wasn’t a problem that could be fixed remotely, I had to call my host and ask them to fix that (they did in a remarkably short time).


Can you hack this out? Hint #1

Here is another challenge, smaller, this time, which can lead you the right path.  And yes, it is a screwy one.

All you have to do is make the following assertion fail.

public class Program
private static void Main(string[] args)

public static void CanThisHappen<T>() where T : class, new()
var instnace = new T();
Debug.Assert(instnace != null,"How did we break the CLR?");

The rules are simple, you are not allowed to do anything outside the current process. That means that profiling API, post compilation assembly rewrite, etc are forbidden. Anything else is a fair game.

I will let you know that I can make the assertion fail in 4 lines of code :-)

Tweets that warm my heart

I got up this morning and the following showed up in my tweeter search:




I would say that from the “is the profiler helping customers?”, we are in a pretty good position.

Can you hack this out? Revised

It appears that I made this challenge a bit too hard, I wrote it in haste and didn’t check that it all works. Here is the new code, it is almost the same as the first, but now it doesn’t check the typeof(T) (for which there is no interception path), but the provided runnerType instead.

internal interface IRunner
void Execute();

public sealed class AssemblyRunner : IRunner
void IRunner.Execute()

public class CompositeRunner
private readonly Type runnerType;

public CompositeRunner(Type runnerType)
this.runnerType = runnerType;

public void Execute()
if (!typeof(IRunner).IsAssignableFrom(runnerType))
throw new InvalidOperationException("invalid type");
var runner = (IRunner)Activator.CreateInstance(runnerType);

Sorry about that image

Look at the future posts for hints for that.

How would you learn a new platform?

Here is an interesting problem that I am facing. I have a pretty good working knowledge of computing, and while I can usually manage to get the gist of a new technology in a short amount of time, that is only useful for talking about it, not actually applying that. I am currently trying to figure out the parts where I am missing, and plug the holes. And I am running into a bit of a problem here.

A case in point, I can read Java, and I can probably write C# code in Java, but I can’t write a Java application. Not for lack of technical skills, but just because I lack the practical knowledge on how to do so.  The problem is that it is going to take too long for me to slog through everything myself, especially since my main problems are in things like IDEs and usage patterns, which aren’t really stuff that you learn from reading about it.

I am thinking on taking a Java course, not so much for the content of the course, as the ability to actually build Java apps in an environment where I can ask questions. On the other hand, picking the right course is problematic, I am not going to sit through “this is a for loop", but I don’t want to be the guy with the blank stare.

The wave experience

I had the chance to take part of a multi people conversation on Google Wave.

Overall, I have to say that it was an interesting chat experience, but it lacks some very important things. Chief among them, if you are going to allow multiple people to edit at the same time and expect to give a real time experience, you have to provide good experience when people are editing “beyond the fold” (the areas of the documents that I am not currently viewing.

Replaying stuff is nice, but it isn’t working for the real time edit portion, which is the most interesting bit.

Overall, I have to say that I found it to be somewhat like IRC, and I dropped out of that years ago, because it had a real time requirements that I could not justify. I much rather use something that is by nature async, with a sync-like experience, like email or twitter.

Uber Prof analysis pipeline

In most situations, I like to think about my apps as a set of composite Lego blocks. Ignoring the infrastructure bits, here is how Uber Prof is structured internally:


I have a set of message processors, which work of low level event messages, detect patterns in those messages and turn them into high level events. Next, I have the Model Building part, which takes the high level information and handle correlation between the different events. I then have analyzers, which are branched out of the model building processor to perform the various analysis actions that I need.

The idea is to try, as much as possible, to build the system out of identical parts, and when I need additional functionality in a given location, I create another extension point and plug in more single use classes at that place.

All of this is being orchestrated by a container that can provide those dependencies, so the act of creating a new class is also the sole act required to add the new feature to the system.


Published at

Does history matters?

In Gmail, right now, I have 65,290 (non spam) messages, stretching back over 3 years. I routinely (once a week or so) need to refer to something that is at least six months old. I just had to dug in old backups to find an instant message log from 2006, because I had some information there that could not be found anywhere else.

My take on that is that yes, history matters, a lot. Moreover, being able to get to that history matters as well.

Today, I cannot imagine how I would try to manage myself without Gmail there to help me, being able to efficiently & easily search my entire history is the number one reason I love it so much.

Oh, and also important, make sure you have the backups :-) That IM log that I was talking about survived about 4 different computer transfers.

Can you hack this out?

I recently had to take a really deep look into how to cheat the CLR, that brought about some interesting discoveries, including the fact that it is, surprisingly, possible to do so.

Let us say that you have this code in some 3rd party assembly that you cannot modify:

internal interface IRunner
void Execute();

public sealed class AssemblyRunner : IRunner
void IRunner.Execute()

public class CompositeRunner<T> where T : new()
public void Execute()
throw new InvalidOperationException("invalid type");
var runner = (IRunner)new T();

Can you get the CompositeRunner to output:

    • starting
    • before executing
    • executing
    • after executing
    • done

The real problem was a bit harder, but this is a good start.

Reducing friction as a standard operating method

image One of the things that I am really sensitive for is friction. And anytime that I run into that, I am doing an evaluation about how much it is going to cost to remove that. Thee are two sides for this evaluation. There is the pure monetary/time side, and then there is the pure opportunities lost side.

The usual evaluation is “it takes 3 minutes to do so manually, and would take a week to resolve in an automatic way”, assuming a standard 40 hours week, we would need to do something for 800 times before it make sense resolve that automatically.

The problem is ignoring the opportunities lost angle. One of the things that most people do not usually consider is that we tend to avoid things that are painful.

If adding content to a website requires you to commit changes to SVN and then log in to a server and update the site, that is about 3 additional minutes that you topped to something that is probably pretty minor. If you are on a slow connection, those can be pretty frustrating 3 minutes. That means that editing the site get batched, so a lot of minor edits are bundled together, until the 3 minutes overhead become insignificant. That means that the site gets updated less often.

If maintaining two profiles of the same application requires that you would merge between slightly different branches (a pretty painful process), they are not going to be in sync. Building a way which allows a single codebase development for multiple profiles is a pretty significant investment, on the other hand. And it introduce complexity into the software.

If committing is going to hit a network resource (and thus take a bit longer), you are going to commit less often. If branching… but you get the point, don’t you?

If releasing your software is a manual process (even to the point of just promoting a CI build to release status), you aren’t going to release very often.

Those are just a few recent examples that I have run to where friction was the deciding factor. In all cases, I put in the time to reduce the friction to the lowest possible level. What I usually look for is the opportunities lost cost, because those tend to be pretty significant.

image I don’t think about releasing the profiler versions, and I released 20 times in the last three days (5 different builds, times 4 different profiles). The build process for the profiler had me write several custom tools (including my own bloody CI server, for crying out load, a daily build site, upload and registration tools, etc). That decision has justified itself a hundred times over, by ensuring that my work isn’t being hampered by release management, and it a very rapid turn around for releases.

That, in turn, means that we get a lot more confidence from customers (we had a problem and it was resolved in 2 hours). There is actually a separate problem here, that we release too often, but I consider that a better alternative than slow response times for customers.

Supporting Linq to Sql as a second target in NH Prof was something that we could do in three days flat. In practice, it took over a month to get to a position where we could properly release it. The time difference was dedicated to doing it right. And no, I am not talking about architectural beauty, or any such thing.

The three days target time for Linq to SQL implies branching the code base and simply making the changes inline. There is relatively very little change from the NH Prof version, and we could just make it all work in a short amount of time.

I am talking about doing it in a way that introduce no additional friction to our work. Right now I am supporting 4 different profiles (NHibernate, Hibernate, Linq to SQL, Entity Framework), and in my wild dreams I am thinking of having 10 or more(!). Frictionless processes means that the additional cost of support each new profile is vastly reduced.

Having different branches for different profiles was considered and almost immediately rejected, it would introduce too much friction, confusion and pain into the development process. Instead, we went with the route on having a single code base that is being branched automatically by the build process. It means that we have a slightly more complex infrastructure, but it also means that we can do it once and touch it only very rarely.

Does it pay off? I would say it does just in terms of not having to consider merging between the different branches whenever I make a change or a bug fix. But it is more than that.

A week ago I didn’t have the Entity Framework Profiler, today it is in a private beta, and in a week or two it is going to go to public beta. All that time spent on integrating Linq to SQL paid itself off when we could just plug in a new profile without really thinking about it.

Linq to Sql profiler – transactions

Well, here is a new thingie, L2SProf can now detect transactions.

Both standard SQL transactions:


And also distributed transactions using System.Transactions:


This is an interesting feature, because you can see how the use of different transaction strategies have a big impact on how you structure your code.

For example, with L2S, it seems like you are encouraged to read outside of a transaction. Something that I, as an NHibernate user, find quite odd.

Reproducing a WPF memory leak

I have run into WPF memory leaks before, and I thought that I fixed them all, but I started getting more reports from people experiencing large memory usage from NH Prof. That lead me to try to track it down again.

Short story, you can get the reproduction here: http://github.com/ayende/wpf-mem-leak

I left an NH Prof instance running overnight, and was able to confirm that there is a memory leak in there, just by seeing how much more memory it had in task manager. One of the nicest things in Windows 7 is the ability to generate a dump directly from task manager.

I loaded that dump into windbg, loaded sos and mscrowks and set out to figure out what was taking so much memory. The most common way to start a memory leak issue is to start figuring out what is taking all this memory.  For that, the !dumpheap –stat command is great, since it give you memory usage by type and size. Here is what I found:

000007fee95b8a88   364674     11669568 MS.Utility.SingleItemList`1[[System.WeakReference, mscorlib]]
000007feeffe4748    51074     13161552 System.Object[]
000007fee77417a8   728189     17476536 MS.Internal.Data.ValueChangedEventArgs
000007fee95aafe8   364133     17478384 MS.Utility.ThreeItemList`1[[System.WeakReference, mscorlib]]
000007fee95b88c0   728814     17491536 MS.Utility.FrugalObjectList`1[[System.WeakReference, mscorlib]]
000007fee95b8838   728814     23322048 System.Windows.WeakEventManager+ListenerList
0000000000496170    19063     28004120      Free
000007fee7741730   728189     40778584 MS.Internal.Data.ValueChangedEventManager+ValueChangedRecord
000007fef0036c90   736210     47117440 System.EventHandler
000007feeffe7a90  1837702     58806464 System.WeakReference

I literally groaned when I saw that, based on my previous experience, it means that the memory leak is generated from inside WPF code, and that I am not likely to fix it. Then I set out to figure out where it is at. My suspicion fell on the statistics feature, it is the only thing that really updates frequently, after all.

So I setup to build a reproduction. Sadly, it is trivially easy to reproduce the issue. Given this XAML:

<ItemsControl ItemsSource="{Binding Path=Statistics}">
<Border Padding="2 0 0 0">
<ColumnDefinition Width="160"/>
<ColumnDefinition Width="Auto" />
<ColumnDefinition Width="*" />
<TextBlock Text="{Binding Path=Key}"
ToolTip="{Binding Path=Key}"
TextTrimming="CharacterEllipsis" />
<GridSplitter Grid.Column="1"
Margin="2 0" />
<TextBlock Text="{Binding Path=Value}"
Grid.Column="2" />

The following code will generate a memory leak:

public partial class Window1 : Window
readonly DispatcherTimer timer = new DispatcherTimer(DispatcherPriority.Normal)
Interval = TimeSpan.FromMilliseconds(1)

private readonly MyModel context;

public Window1()
context = new MyModel();
timer.Tick += (sender, args) => context.UpdateData();
DataContext = context;

public class MyModel
public ObservableCollection<DictionaryEntry> Statistics { get; set; }

public MyModel()
Statistics = new ObservableCollection<DictionaryEntry>();

public void UpdateData()
Statistics.Add(new DictionaryEntry("Time", DateTime.Now.ToString()));
Statistics.Add(new DictionaryEntry("Mem", GC.GetTotalMemory(true).ToString("#,#")));

When you run the code, you can see that memory usage is going up all the time. As far as I can tell, I am not doing anything that is wrong or even slightly strange here. You can get the reproduction here: http://github.com/ayende/wpf-mem-leak

I tried several workarounds so far, but I didn’t like any of them. Any ideas?

NHibernate Course – Aarhus, Denmark @ Dec 09

There are still available places for my 3 days course about NHibernate in Aarhus on December 9th.

This course is aimed at developers who are new to NHibernate or those who want to amp up their NHibernate knowledge. In the course, we go over mapping options, queries and optimizations, caching, extensibility options and a whole lot more.

I gave this course several times already, and it is pretty intense, but at the end of it, you are going to come out knowing NHibernate.

You can register to the course here:


L2S Prof hardships – getting query duration and row counts

I really want to be able to provide duration & row count for Linq to Sql. It has became a personal goal of mine.

The problem is that there is absolutely no open extensibility points within L2S that would allow it. The solution is… to  break down the wall. If you don’t have extensibility points, then I will make some.


And, you know what? I did:


The problem is that it took almost 12 hours to get it to this point, and along the way I had to build a new provider for Linq to Sql. And yes, that is not possible. Not isn’t supposed to be possible, just is not possible.

To get this to work I had to build a hack on top of a crack on top of a very angry duck.


I have been working on this for the last 12 hours, and this is about the most unsupported method that you could find to hook into things (I had to break the CLR type system to get this to work, and the things that I do with reflection are probably amoral in at least three different religions).

The problem with going with that route is that now I have to delay the public beta for L2S Prof. The good thing is that I can now provide additional information about Linq to SQL.

NH Prof New Feature: Exporting Reports

I mentioned before that exporting reports (so you can mail them around) is something that I would really like to do. In fact, I decided to put it out as a challenge, and I got some really great responses.

Josh Perry has been able to flat out amaze me with a solution that was at once simple, elegant and beautiful. It is an HTML report that contains all the styling inline (so it supports emailing it around, for example, to the DBA) but it also has some java script smarts around it that make it behave like the real application.

As a matter of fact, I have added the exported version as a “live demo” for NH Prof.

I am quite happy with this. You can export your NH Prof load using File > Export, giving you a single HTML that is easily achievable, mailable, etc.

 Check it out.

Some observations on Linq to Sql & Entity Framework codebases

So, I have to do a lot of L2S and EF work recently, while trying to create L2SProf and EFProf. The interesting tidbit is that I got a totally different approach to using them than almost anyone else.

I don’t really care for either as an OR/M (I have my own, thank you very much), I only care for plugging into them deeply enough that I can get the information that I want.

With L2S, that was fairly easy, all I needed to do was figure out how to get access to the logging that it was already doing, and I was pretty much done. With EF, there isn’t any logging, and I was left having to write my own EF provider (and then plugging that in). I am going to talk about how I plugged myself into EF at some length some other time.

One of the most frustrating things with L2S is while the initial implementation was significantly easier than EF, it doesn’t take very long to hit a hard wall of internal stuff. What is worse, it is pretty obvious that there was some pretty sweet architecture inside that got totally killed for non technical reasons. Sometimes, I feel like a code archeologist. I mean, take a look at SqlProvider.Initialize() in reflector for a second:


I am willing to bet that this stuff was added after the decision to kill the L2S provider model was added. This kind of stuff really need to be in a separate provider, but that would have forced IProvider to be public, so it was hacked around.

With the EF version, there are absolutely no visibility into what EF is doing (by that I mean logging, tracing, etc). It does have extensibility in it, however, which means that I could plug into that and then replace enough things until I had the hooks in place to provide the information that I needed. This isn’t that much fun, I’ll admit, but it is working for me so far.

The initial leap to get EF profiling working is much harder, but once that is done, it is pretty straightforward to get most of the stuff that I need. L2S basic architecture make the first few steps very easy, but then you get into a maze of blocked internal stuff.

The more I work on this, the more I like this blog’s tagline.

Non invasive Windows event tracing listening?

One of the things that I would like to do with L2S Prof is to figure out how to detect row counts and query duration. The problem is that there is absolutely no way of doing that at the Linq to SQL level.

So one of the things that I have been researching is trying to get to the tracing that are present at the System.Data.SqlClient level. For more information about those, you can read this article, which explains them in depth. Basically, Windows comes with a really good support for optional tracing, and System.Data.SqlClient makes use of that in order to trace out a lot of information, including what I would need in order to provide row counts and query duration.

The problem?

Enabling this support is a very invasive process, requiring multiple steps and affecting the entire machine. What I would really like to do is to find some way of tracing those events by flipping a switch inside the application that I am running, and then gathering the interesting stuff inside the same application. I don’t want to muck around with machine global settings.

Right now, I don’t see any way of doing that, so I think that I am going to leave it as that for now.

If anyone can point me to how to do it, I’ll add those capabilities to L2S Prof, but until that happens, I am afraid they aren’t going to be there :-(

Linq to Sql Profiler is now on public beta


Well, after talking about it quite often recently, I think it is just about time to make this public.

The Linq to Sql Profiler is now on public beta, and it is even more awesome than you could imagine.

We are able to get a tremendous amount of information out of Linq to Sql, so from profiling behavior it is quite on par with NH Prof or Hibernate Profiler.

Here is a screen shot:


Please note that the UI color scheme is still under development, we would love to hear feedback about that as well.

Like NH Prof, L2S Prof is currently offered in a 30% discount for the duration of the beta period.

Looking forward for your feedback…

Debugging Win32 crash in NH Prof

One of the things that happened when I pushed the UberProf build of NH Prof out is that people started complaining that trying to load a saved snapshot would crash the application. It took me a while to reproduce that, but I finally manage to get this:


When I tried to debug it, all I got was this, and native stack trace that I had no way of actually resolving.


Trying to reproduce the problem in the debugger resulted in the same experience. I tried playing around with WinDbg for a while, but I am not very good at that, so I gave up and tried something that I find useful in the past. Tell Visual Studio that I want it to capture all types of exceptions, not just CLR exceptions:


Using this mode, I managed to get:


This led me to inspect the CancellableWaitBoxView.xaml, which contains:


And now it all made sense. With the UberProf builds, each profiler has a distinct assembly name. The problem is that this XAML refer to the master assembly name (which I use for development). 

The reason this issue was hard to fix was that it is WPF window failing, and then somehow that exception is being swallowed and translated into a native error dialog, I am not quite sure why. Once I knew what the problem was, the fix was quite simple (using assembly resolve).

Would you just take my money please?

I just finished doing my reverse bills. Unlike doing my bills (which are usually handled automatically), this is tracking down who I owe money to that haven’t been paid. I just had three different incidents where I had to call up people and remind them that I probably need to pay them money.

Some highlights from those conversations:

  • Hey, I owe you money, can I pay it already?
  • What do you mean, I don’t owe you money. I say I do, and the customer is always right.
  • You promised that you would send me that invoice, but it isn’t here… What do you mean, it is in the mail?

It feel funny, because some of those talks went along the line of “you own me money”, but what I was trying to get is just an invoice so I could actually pay it. Hell, I had to call my landlord and setup time & place so I can pay for utilities (he has been avoiding that for the last few months). And the last rent check I had to ring him up and ask him why he didn’t deposit the check he had.

It is a funny world we are living at.

On the one hand, I don’t feel it is ethical not to pay, or even to wait until they wake up and remember that they need to be paid. On the other hand, I like that there is enough of a trust that this situation can actually come up.

Chasing the SQL Injection that never was

So, I am sitting there quietly trying to get EF Prof to work in a way that I actually like, when all of a sudden I realize that I am missing something very important, I can’t see the generated queries parameters in the profiler.

Looking closely, I started investigating what appear to be a possible SQL injection issue with EF. My issue was that this query:

entities.Posts.Where(x=>x.Title == “hello”)

Generated the following SQL:

1 AS [C1],
[Extent1].[Id] AS [Id],
[Extent1].[Title] AS [Title],
[Extent1].[Text] AS [Text],
[Extent1].[PostedAt] AS [PostedAt],
[Extent1].[BlogId] AS [BlogId],
[Extent1].[UserId] AS [UserId]
FROM [dbo].[Posts] AS [Extent1]
WHERE N'hello' = [Extent1].[Title]

It literally drove me crazy. Eventually I tried this query:

var hello = "hello";

Which generated:

1 AS [C1],
[Extent1].[Id] AS [Id],
[Extent1].[Title] AS [Title],
[Extent1].[Text] AS [Text],
[Extent1].[PostedAt] AS [PostedAt],
[Extent1].[BlogId] AS [BlogId],
[Extent1].[UserId] AS [UserId]
FROM [dbo].[Posts] AS [Extent1]
WHERE [Extent1].[Title] = @p__linq__1

This was more like it.

It seems (and Julie Lerman confirmed it) that EF is sticking constant expressions directly into the SQL, and treating parameters differently.

I am not quite sure why, but from security standpoint, it is obviously not a problem if it does so for constants. It have a lot less hair now, though.

Über Prof goes live

Über Prof is the codename for the multi-OR/M profiler that I have been working on for the last several weeks.

Today is a good milestone, because it marks the shift from treating it as an experimental to production ready.


Über Prof also brings with it several changes to the way I mange the profiler profiles. Each profile (NHProf, HProf, L2SProf and EFProf, currently) is built on top of the same codebase, but contains slightly different functionality to fits it target audience. I might talk about exactly how we are doing that in a later post.

That required making a small number of breaking changes:

  • The executable filename was changed to NHProf.exe (or HProf.exe, or L2SProf.exe, etc)
  • The appender filename was changed to HibernatingRhinos.Profiler.Appender.dll
  • The appender type name was changed to HibernatingRhinos.Profiler.Appender.[profile].[profile]Profiler.Initialize();

In practice, those changes are annoying, but should be very easy to fix.

Something important to note, however, is that NHibernate profiling and Hibernate profiling has been split.

For NHibernate, we have NHProf, and for Hibernate, we have HProf.

I hope to have L2sProf out in public beta this week. And EFProf is currently in private beta for this week and hopefully poke its nose out as public beta next week.


  • There is going to be a bundle, yes. No details yet on what it will be, though. I am leaning more toward a discount model rather than a bundle, since it seems that it would be more appropriate.
  • People who already bought NHProf license can use that for HProf as well with no issues.

Answering YAGNI commentary

My post about applying YAGNI in Impleo has gotten a lot of comments, and instead of answering them one at a time, I think it would be useful to just have a post answering all of them together.

Ryan Riley: I suppose I'm still wondering, how did WebForms appear as a result of YAGNI? I realize that's the original ASP.NET platform, but MVC seems simpler to me, and even simpler would be starting with HTML, CSS, and JavaScript (or, heaven forbid, classic ASP) with some AJAX.

Ryan, it is quite simple. It is all about friction, using HTML, CSS & Javascript would have added more friction initially than the webforms solution.

Torkel: Any plans to open source it?

Highly unlikely.

Andrew: I know we all hate Web Forms, yada yada yada, but if you need to just setup a few pages, it's dead easy and much faster than MVC, a WCF service or any other method to deliver content.  Hence why it makes perfect sense in this "I'm using YAGNI to the letter of the law" experiment.
But the reality is, you rarely have a project where you can work in this pure YAGNI way, since there almost always is requirements such as make it testible, make it easy to maintain, make it SOE friendly, etc. so working in ASP.NET Webforms a poor choice for the job.  But there does come a time where a project is "All we need is one page that shows data in a grid", it can be done in literally 10 minutes with a WebForm.

I agree, except that there is an additional thing that I want to bring to the table. The most important thing for me in most of my apps is the first customer demo. That is important for a host of reasons, but most importantly, because it make a product real. Building fancy architecture and buzzward compliant development is usually in the way of that. The first customer demo is simple, stupid and should be out as soon as possible.

Dmitriy Nagirnyak:  

Your own website framework?
What are the reason for starting it from scratch if there are some of them already available?
- Exercise?
- Lack of extensibility?
- Wrong architecture?
- "Just want my own" thing?
- others?

Quite simple, I wanted something that would work the way I need it to work. Anything that adds friction to the process is not acceptable. I am willing to build my own thing to get a friction free process.

Richard Dingwall: Article starts with YAGNI and ends up writing a whole new CMS from scratch! Ayende!!

See previous reply.