Ayende @ Rahien

Refunds available at head office

Creating objects - round 2

It was pointed out that I had a skew in my test, I was also calling i.ToString() in a tight loop, which probably kills the numbers.

Here is the exact same benchmark, but using a constant string value, instead of calling i.ToString() all the time.

  • Using new - 00.0177508 seconds (down from 00.3648117 seconds)
  • Using Activator.CreateInstance - 06.3033382 seconds (down from 06.8242636 seconds)
  • Using GetUninitializedObject - 02.8209057 seconds (down from 03.2422335 seconds)
  • Using specialized dynamic method - 00.0417958 seconds (down from 00.4314517 seconds)
  • Using generic dynamic method -  00.1189762 seconds (down from 00.5018288 seconds).

Here are some numbers to keep you entertained.

Difference from previous approach Time to create single instance Iterations Time Method
0% 0.0000000177508 1000000
0.01775
New
235.46% 0.0000000417958 1000000
0.0418
Specialized dynamic method
284.66% 0.0000001189762 1000000
0.11898
Generic dynamic method
2370.98% 0.0000028209057 1000000
2.82091
GetUnitializedObject
223.45% 0.0000063033382 1000000
6.30334
Activator

It doesn't change the overall conclusion, by the way. There is just no way I can make myself work about 0.0000063 second.

I was also asked what the cost of creating dynamic method is, it comes to the tune of 00.0012487 seconds, for the generic version that I have shown. I would strongly encourage caching that, since (relative to creating the object itself) is seems expensive.

It is important to remember that dynamic method are also garbage collected, however, so we don't really have to worry about blowing the AppDomain if we use them.

The mysterious life of mutexes

Last night I was talking with Ron Grabowski (of log4net fame) and the subject of mutexes came up. Specifically, the subject of abandoned mutexes across threads, AppDomains, and Processes. I was pretty sure what the behavior should be, but we wrote a set of test cases for that, with... surprising results.

My expectation that after I get a hold on a mutex, I have to explicitly let it go. If I am killed or interrupted somehow, then I would expect the next person to acquire the mutex to get an AbandonedMutexException.

The first code sample does exactly that:

new Thread(delegate()
{
    Mutex m = new Mutex(true, "foo");
    m.WaitOne();
    Console.WriteLine("Got mutex");
    Thread.Sleep(3000);
    Console.WriteLine("Bye..");
}).Start();
 
Thread.Sleep(500);// give the thread some time to get the mutex
Mutex m2 = new Mutex(true, "foo");
m2.WaitOne();

We haven't explicitly release the mutex, but we let the thread die, on acquire, you'll get the expected AME.

Now, let us try it with processes, shall we?

if (args.Length != 0)
{
    Mutex m = new Mutex(true, "foo");
    m.WaitOne();
    Console.WriteLine("Got mutex");
    Thread.Sleep(1000);
    Console.WriteLine("Bye, fast");
    Environment.FailFast("blah");
}
else
{
    Process p = Process.Start(typeof(Program).Assembly.Location, "a");
    Thread.Sleep(2000);
    Mutex m = new Mutex(true, "foo");
    m.WaitOne();
    Console.WriteLine("got it second");
}

We start our process, which spawn a second process, which would acquire a lock on the mutex, and then fail fast, killing the process. The first process, on attempting to get the process, will die horribly, as expected.

But not so fast, because this bit of code doesn't throw anything, and pretend that everything is right.

if (args.Length != 0)
{
    Mutex m = new Mutex(true, "foo");
    m.WaitOne();
    Console.WriteLine("Got mutex");
    Thread.Sleep(1000000);
}
else
{
    Process p = Process.Start(typeof(Program).Assembly.Location, "a");
    Thread.Sleep(2000);
    p.Kill();
    Mutex m = new Mutex(true, "foo");
    m.WaitOne();
    Console.WriteLine("got it second");
}

I am not really sure why this is the case, I would expect Envirionment.FailFast and an external Process.Kill to have the same behavior.

It gets to be even stranger, because we see the exact same problem when we are dealing with AppDomains:

public class Foo : MarshalByRefObject
{
    public void Repeat()
    {
        bool ignored;
        Mutex m = new Mutex(true, "blah", out ignored);
        m.WaitOne();
        while (1 != DateTime.Now.Ticks)
        {
            Console.Write(".");
            Thread.Sleep(1000);
        }
    }
}
 
AppDomain app = AppDomain.CreateDomain("fo");
new Thread(delegate()
{
    Foo f = (Foo)app.CreateInstanceAndUnwrap(
        typeof(Foo).Assembly.FullName,
        typeof(Foo).FullName);
    f.Repeat();
}).Start();
Thread.Sleep(1000);
bool ignored;
Mutex m3 = new Mutex(true, "blah", out ignored);
Console.Write("aquiring");
m3.WaitOne(TimeSpan.FromSeconds(2), false);
AppDomain.Unload(app);
Console.Write("done");

I am not willing to comment on what is going on, I have the feeling that I am missing something.

Nevertheless, this is surprising behavior.

Continuous Environment Validation

For some reason, it seems like I am talking about Release It a lot lately, to a lot of people. As I said when I reviewed it, that book literally changed the way that I approach problems. It also made me much more aware of the failure scenarios that I need to deal with.

A while ago I sat down in one of Jeremy Miller's talks and he mentioned that he had added the ability to do Environment Validation to StructureMap, so when the application is starting up, it can verify that all its dependencies are in a valid state. That made so much sense to me that I immediately added this facility to Windsor.

What I am going to talk about today is to take this approach one step further. Instead of running those tests just at application startup, they should be run every day, or every hour.

Yes, the operation team is suppose to have monitoring on the application, but unless they were part of the development process (or are a dedicated ops team), that still leaves you as the principal owner of knowledge in about the environment your application need. Even if you have a capable operation team, and they have very good understanding on your application, it is often best to support them by providing this functionality. It is very likely that you can get more information from your application that the operation team.

And if you don't have an operation team, you really want to be able to do that.

Now that we have taken care of the motivation for this approach, let us see what exactly we are talking about.

Environment validation means that you validate the your entire environment is in a state that allows your application to run in full capacity. I am going to list a few things that I think are essential for many applications, I am sure that I am going to miss some, however, feel free to add more items to the list.

  • Certificate's valid and expire in more than a month.
  • Domain registration expires in than one month.
  • For each server in the application (web, database, cache, application):
    • Server is alive and responding (within specified time).
    • Server's HD has more than 10% free space.
    • Server CPU usage is less than 80%
  • Associated 3rd party servers are responding within their SLA.
  • Sample execution of common scenarios finish successfully in a specified time frame.
  • Number of faults (non critical ones) in the application is below the threshold.
  • No critical fault (critical defined as taking the entire system down).
  • Current traffic / work on the system is within expected range (too low, and we may have external network issue, too high, and we need to up our capacity).
  • Application audit trail is updated. (Can do the same for log, if required).
  • System backup was performed and completed successfully.
  • All batch jobs have been run and completed successfully.
  • Verify the previously generated faults has been dealt with.

Those are the generalities, I am pretty sure that you can think of a lot more that fit your own systems.

The important thing to remember here is that you should treat this piece as a core part of the application infrastructure. In many production environment, you simply cannot get access. This is part of the application, and should be deployed with the application. At any rate, it should be made clear that this is part of the deployment program, not just useless appendix.

My preference would be to have a windows service to monitor my systems and alert when there are failures.

This is another important consideration, how do you send alerts? And when? You should have at least three levels of warnings: Warning, Error and Fatal. You send them according to the severity of the problem.

In all cases, I would log them to the event log at a minimum, probably send mail as well. For Error and Fatal levels, I would use SMS / generate alert to operation monitoring systems. If there are monitoring system in place that the operations team is using, it is best to route things through them. They probably have the ability to wake someone up in 3 AM already. If you don't have that, than an SMS is at least near instantaneous, and you can more or less rely on that to be read.

That is long enough, and I have to do some work today, so I'll just stop here, I think.

Creating objects - Perf implications

Here are a few examples of how we can create objects, and the perf implications of each way. In all those tests, I have used the following class as my benchmark.

public class Created
{
    public int Num;
    public string Name;
 
    public Created(int num, string name)
    {
        this.Num = num;
        this.Name = name;
    }
}

We have a value type and a reference type that are passed to the constructor.

The test code is here:

static void Main(string[] args)
{
    int iterations = 1000000;
    Stopwatch watch = Stopwatch.StartNew();
    for (int i = 0; i < iterations; i++)
    {
        CreateInstance(i);
    }
    Console.WriteLine(watch.Elapsed);
}

Here it my base line, calling the constructor directly.

private static Created CreateInstance(int i)
{
    return new Created(i, i.ToString());
}

This executes in 00.3648117 seconds on severely underpower laptop. Pretty good, considering we just created a million instances. Now, let us see what happens if we user Activator, shall we?

private static Created CreateInstance(int i)
{
    return (Created)Activator.CreateInstance(
                        typeof(Created), i, i.ToString());
}

This run depressingly slow, 06.8242636 seconds.

Let us try to improve that a bit, using GetUninitializedObject and directly invoking the constructor.

static ConstructorInfo ctor = typeof(Created).GetConstructors()[0];
private static Created CreateInstance(int i)
{
    object o = FormatterServices.GetUninitializedObject(typeof(Created));
    return (Created)ctor.Invoke(o, new object[]{i, i.ToString()});
}

This runs in 03.2422335 seconds, a significant improvement, but we can do even more, I think.

We start by making the required definitions:

static ConstructorInfo ctor = typeof(Created).GetConstructors()[0];
delegate Created CreateCtor(int i, string s);
static CreateCtor createdCtorDelegate;

Then we generate a dynamic method to create the object, and turn that into a delegate:

DynamicMethod method = new DynamicMethod("CreateIntance", typeof(Created),
    new Type[] { typeof(int), typeof(string) });
ILGenerator gen = method.GetILGenerator();
gen.Emit(OpCodes.Ldarg_0);// i
gen.Emit(OpCodes.Ldarg_1);// s
gen.Emit(OpCodes.Newobj, ctor);// new Created
gen.Emit(OpCodes.Ret);
createdCtorDelegate = (CreateCtor)method.CreateDelegate(typeof(CreateCtor));

Now, in the method itself, all we have to do is call this delegate:

private static Created CreateInstance(int i)
{
    return createdCtorDelegate(i, i.ToString());
}

And now that run very fast... 00.4314517 seconds. Almost as fast as our baseline. But this is not really a good example, I am afraid. At least, not a god example of generally creating instances, let us make this into the more general form, shall we?

We will change the CreateCtor delegate to the following signature:

delegate object CreateCtor(object[] args);

And the generation of the dynamic method to use the generic approach:

DynamicMethod method = new DynamicMethod("CreateIntance", typeof(Created),
    new Type[] { typeof(object[]) });
ILGenerator gen = method.GetILGenerator();
gen.Emit(OpCodes.Ldarg_0);//arr
gen.Emit(OpCodes.Ldc_I4_0);
gen.Emit(OpCodes.Ldelem_Ref);
gen.Emit(OpCodes.Unbox_Any, typeof(int));
gen.Emit(OpCodes.Ldarg_0);//arr
gen.Emit(OpCodes.Ldc_I4_1);
gen.Emit(OpCodes.Ldelem_Ref);
gen.Emit(OpCodes.Castclass, typeof(string));
gen.Emit(OpCodes.Newobj, ctor);// new Created
gen.Emit(OpCodes.Ret);
createdCtorDelegate = (CreateCtor)method.CreateDelegate(typeof(CreateCtor));

We need to make the following modification to our CreateInstance method:

private static Created CreateInstance(int i)
{
    return (Created)createdCtorDelegate(new object[]{i, i.ToString()});
}

And now it runs in.... 00.5018288.

Now, what does this long and arcane post tells us?

Creating instances, no matter how many, is really cheap. Remember that I had to do a million iteration to get something measurable.

The table bellow give the final statistics. Pay attention to the last column, this gives the amount of time it take to create a single instance. Even the most hard core perf fanatic would be hard pressed to argue over 0.000007 seconds. At least I hope so :-)

Create single instance Iterations Time Method
0.00000036481170000000 1000000
0.36481
new
0.00000682426360000000 1000000
6.82426
Activator
0.00000324223350000000 1000000
3.24223
GetUninitializedObject
0.00000050182880000000 1000000
0.50183
Dynamic Method

And just to clarify, you are not going to see anything like creating a million object in most scenarios that you care about. In other words, you can probably leave things well enough alone.

WCF, Federated Security and Custom Authentication Token, oh my!

So today I spiked some code to see how hard it was to get federated security to work using WCF. I have never had to do something like that before, but I am familiar with the concepts, at least.

Armed with the WCF federation sample, I set out to build my own.

I failed.

I cannot believe how complex this was. I stopped after I realized that my app.config file passed 200 lines, solely for WCF configuration, I got a lot of unhelpful exceptions from WCF and generally felt that I am wasting my time.

Then I turned to building a custom authentication token, that would have served my need well enough. I started to follow the steps in the sample, but by the fifth class that I had to implement, I had enough. I started counting the number of classes that are required to build a custom authentication token. 11(!) classes!

Sorry, at this point, I back away slowly. This is way too complex for what I need. I solved the problem using a message header, which was a sweet & simple solution to the problem.

I consider myself fairly adept at handling complexity, but oh my god, just trying to touch this piece was scary on many levels.

Distributed cache for the CLR

Sriram has just published a post about Cacheman, a pet project  his that give us a Memcached like functionality based on the CLR.

I started a project like that a while ago, but eventually I decided that it would be much easier to just use Memcached. Some of the tricks that Memcached is using is explicit memory management of the layout of the cache. I assume that the GC will take care of much of that for us.

Sriram has managed to get to 16,000 requests / second with his current code. Which is certainly impressive.

I find it very interesting because I am much more comfortable with CLR based tools. I know their failure scenarios and how to deal with them. Native tools... not so much.

I have used Memcached for production, and it works. But assuming that Cacheman would be on par with the Memcached implementation, I believe that I will choose it.

I am reading the code now (well, reflecting over it) and it is interesting read.

Zero Friction and why Defaults Matters

Several times recently I had to answer why I care so much for defaults if they are something that I can change. My answer is usually a derivative of "defaults matters". I have come to understand that this is not a sufficient answer and I need to clarify it.

I have talked extensively about Zero Friction environment and why I care so much about it. It is about how much I have to fight the tools that I have in order to get what I want.

I spent much of yesterday moving Rhino Tools from MsBuild to NAnt, because MsBuild doesn't have any way to get the current framework version. That is a big task, and considering that I can add this functionality to MsBuild in a few minutes, why did I do that?

Again, it had to do with the fact that when I need a solution, I had to work hard for it. With NAnt, it is just there. For most of everything, in NAnt, it is just there. Between the two, NAnt is the tool with less friction. In the long run, this mean that I would be able to work more efficiently.

How does this related to defaults? Well, consider this, if I have to extend / change something to get something that I consider a baseline functionality, this is a problem. This is friction, and if it generating that much friction early on, I can safely assume that it will generate much more friction when I want to do the advance stuff.

At that point, thanks but no thanks.

To put it in more familiar terms, whatever defaults you have are the Out Of The Box Experience that you have for your software. If you missed the default, it is as if you just hand me a product in one of those ugly and hard plastic shell (the ones that you need scissors just to open) and told me that I can put it in a pretty designer box.

Yes, I can, but you missed your chance to get me.

And that is ignoring the fact that default, for most people, are what is happening. You need good defaults.

Generic extension methods

I was playing around with the compiler when I hit this interesting feature. I was very surprised to see that this has compiled successfully.

   1: static class Program
   2: {
   3:     static void Main(string[] args)
   4:     {
   5:         IProcesser<GZipStream> p = null;
   6:         p.HasTimeout();
   7:     }
   8: }
   9:  
  10: public static class Extensions
  11: {
  12:     public static bool HasTimeout<T>(this IProcesser<T> s)
  13:         where T : Stream
  14:     {
  15:         return s.Desitnation.CanTimeout;
  16:     }
  17: }
  18:  
  19: public interface IProcesser<TDestination>
  20:     where TDestination : Stream
  21: {
  22:     TDestination Desitnation { get; }
  23: }

MsBuild vs. NAnt

A long while ago I moved the Rhino Tools repository to using MSBuild. I am not quite sure what the reason for that was. I think it was related to the ability to build it without any external dependencies. That is not a real consideration anymore, but I kept that up because it didn't cause any pain.

Now, it does.

More specifically, I can't build the rhino tools project on a machine that only have 3.5 on it. The reason is that I have a hard coded path with 2.0, which worked well enough, until I tried to work on 3.5 machine. In which case all hell broke loose.

Well, that is an easy fix, right? Just go and change the hard coded 2.0 to the current framework version, right?

Except that the msbuild doesn't have any way to do that.

I am surprised, to say the least. And yes, I know that I can extend it to support this. I am not particularly sure that I want to, however. At this point, it would be easier to just move it to NAnt, I think. It would also align rhino tools with the tooling all its dependencies are using.

Reviewing Unity

I am sitting at the airport at the moment, having to burn some time before I can get on a flight, and I decided to make use of this time to review the Unity container from the P&P group.

A few things before I start. Unity is a CTP at the moment, it is not done, and that is something that should be taken into account. I have several design opinions that conflict with the decisions that were made for Unity. I am a contributor to Windsor. Overall, I am biased. Please take that into account.

I am also going to put it through some of the paces that I put Windsor through. Just to see how it goes. Again, I am comparing a > 4 years old container to a very new new, another thing to keep in mind. I do not foresee a situation in which I will use Unity over Windsor, but I do hope that this post will be helpful in future improvements to Unity.

A lot of this feedback is based on my experience and may contain comments that can be considered personal preferences.

Let us start with the things that I like:

  • This piece of code is very nice, and it is useful in many scenarios where the instance is created by an external source (Page instance, Forms, Web Services, and many more). At the moment, this is something that Windsor doesn't have (I need two find a few hours free on a machine that has a development environment.
    myContainer.BuildUp<MyRealObject>(myObject);
  • I am ambivalent as far as the Method Injection goes, in the abstract, I say cool, nice feature. The practical side says that it is mostly not required. Nevertheless, let us put it here.
  • The XML configuration is very clean and easy to follow. At least the sample ones that I have seen. I am not sure if it is possible to put it in an external file (not in app.config), but that is probably my only reservation with that.
  • The ability to configure instances from the configuration is cute. Especially if you add the converter support that you have there.  It doesn't seem, at least from a brief glance, that you can build complex objects there, though. That is not a bad thing, however :-)
  • It was a really good decision, to decouple the container and its configuration.

And now to the things that I don't like:

  • The documentation (and a feature which attempts to resolve a type by default, even if it is not registered) direct toward having concrete class dependencies. I strongly dislike this, I think that except for rare cases, an interface should be used. And I generally dislike this feature.
  • For that matter, the ability to resolve a type that wasn't registered in the container strikes me as problematic. There were some long discussions on that in the ALT.Net mailing list, and I can see the other side's point of view. Yes, the container should enable this feature, but not in the common scenario. Preferably, add a method that make it explicit what you are doing.
  • According to the documentation, having several constructors will result if Unity picking one of them at random, and supplying its dependencies. I find the greediest possible ctor approach much more predictable, and a more natural model.
  • [DependencyConstructor] bothers me. This is the container invading into the application in an unseemly manner. It bothers me more that it is not optional if you have more than a single ctor.
  • Much worse, from the point of view of container ignorance, is the facilities for property injection. Here you have to specify [Dependency] on all the properties that you want injected. Again, this is the container reaching into my code and messing with that in unseemly ways. My code should have nothing to do with the container whatsoever. By the shape of my object, so will the container work. Not the other way around.
  • container.Get<ISomethingMadeUp>() will return a null reference if it can't create the type. I think that this is the wrong design decision. It should definitely throw here. It means that instead of getting a nice exception "ISomethingMadeUp is not registered" I get "Object not set to an instance of an object". While the ability to try resolve from the container is useful, it should be in a separate method. TryGet would be my preference for that.
  • Introspection capabilities are missing. That is, there is no way (that I could easily see, and I looked for one, to ask if a given type or a given extension are registered in the container. This is critical for many advance scenarios.
  • Something that I consider to be a sever bug, but from the code looks like a design decision, is ignoring missing dependencies when you resolve a component. That is, assume that you component Foo that have constructor dependency on ILogger. If you resolve Foo, and the container will not be able to find an ILogger implementation, it will pass null to the constructor. Read that one again. Instead of getting "Cannot resolve Foo because ILogger cannot be constructed" you are going to get "Object reference not set to an instance of an object".
    This is a problem, period.
  • Yes, you can use [Dependency(NotPresentBehavior)] to change that, but that is again, an invasive approach. Not a solution.
  • There is no protection from cycles in the dependency graph A depends of B depends on C depends on A. Again, you need to get a good exception here, explaining the problem. What you get is a stack overflow exception and bye bye to the program.
  • No support for generic components. That is, registering IFoo<> and getting IFoo<string>.
  • At a certain point, calling a method on the container will raise an event that will be handled by a strategy. Is seems like a very odd thing to me. The case in point is setting a component to be a singleton, I would have grabbed the singleton strategy directly and called that, instead of going through the event.
  • Life time support. Right now Unity supports two levels, singleton and transient. That is good, and you can probably extend that with policies. The problem is that those ideas are hard coded into the container interface. Off hand, you need at least local (per thread or per request) models as well. And I think that exposing singleton at the container interface is a mistake.
  • I haven't seen how you can control the dependencies yet, which is a very important concern. That is, when I am building EmailSender, I want it to use port 435 and the file logger, instead of the email logger that everything else uses. This is a key feature, but reading the forums, this is currently a problematic feature.

Object Builder 2 Observations:

  • I like the use of dynamic methods to create instances, but the implementation (at least at first glance) seems awfully complicated. The main object that I have here is that you have the control flow of generating this method separated over a large number of objects and interfaces. The advantage that this give you is fine grain control over this process, but consider the task involved, this is a highly cohesive process, and you can't really just plug your own stuff to it without properly understanding everything that goes there.
  • In the same vein, the use of policies and strategies for everything obscure the intent of the code. The DynamicMethodConstructorStrategy, for example, need to get the generation context, the existing parameter is use for that, which seems like a bad abuse of the given API. I would create an independent component with an API that would explicitly make this dependency. If you wanted the ability to replace that, have it registered in the container by default, then replace that before you resolve anything.
  • In the above case specifically, we have the existing parameter that sometimes holds the existing object, and sometimes holds the current context. That is not a friendly API.

I wanted to try Unity's extensibility model, but I think that I put enough time in it already. What I have learned so far makes it pretty much pointless.

I wanted to build the StartableExtension, which automatically starts up components that implements the IStartable interface. On the surface, it is very simple, register to TypeRegister event, get the type and Start() it. The problem is that you need to wait until all the type's dependencies are available. For example, I may register IHealthChecker before I register ILogger, which it need. There is no way in Unity at the moment to enable this scenario.

Something else that I would like to see is Unity's version for Environment Validation. Here is is likely to be possible to do this, but it expose a problem with Unity's dependency model. You can't register arguments for a component, you can only do that globally, and that is if you are willing to perform invasive operation on your components.

Overall, I think that is an okay solution. You can see some Windsor influence in the API and in the configuration, which make me happy.

It has too many red lines there for me to be able to use it (even if Windsor wasn't around), however. Some of the things that I have outlined here (error handling, for example) can be fixed easily enough, I am much more concerned with the design decisions that were made. Specifically, not throwing when a dependency is not found, both in the specific case of directly calling Get() and in the general case of resolving a dependency for a type that is currently being resolved.

The invasiveness of the container is one of the top issues that I would have fixed. That is one of the major reasons that CAB got lambasted.

I remember having to work with Object Builder circa 2005 - 2006, and it was a major pain. Unity looks significantly better than that. As well as the bits of Object Builder 2 that I have examined. I still object to the strategies / policies design, however, it seems like an overly generic solution. And methods like AddNew offend me for some reason. Add( new Foo() ) isn't hard.

And, to save myself a lot of trouble later on, this is not an attack on Unity, this is simply me going over what is there and expressing my professional opinion on an early version of a project.

Ayende's Design Guidelines: Rule #1

When creating a generic method, strongly prefer to supply an overload that is not generic, can accept the types manually, and is externally visible.

Reasoning: The generic version often looks significantly better than the non generic version, but it comes with a cost. It assumes that you know, up front, what the types are. When you are writing any type of generic code, this is almost always not the case and your generic method is useless in that scenario.

Example: myContainer.RegisterComponent<IService, ServiceImpl>(); is a good syntax to have, but the problem with that is that it cannot be used with this code:

foreach(Type type in GetComponentTypes())
{
    myContainer.RegisterComponent<type.GetInterfaces()[0], type>();
}

Since we cannot use the generic overload, we need to resort to trickery such as MakeGenericMethod and friends. This is costly at runtime, obscure and generally make life harder all around.

Handling dependencies in a one assembly

There were several comments for my recent post about the disadvantages of creating a single assembly for a project, instead of the more common multiply ones.

The main one seems to be related to dependency management. How you can be sure that the UI isn't accessing the database, and other fun violations of layering.

I have to say that I don't really have much issue with that. I want good layering, but I find that code style in a project tend to keep consistent over a long period of time in any reasonable time. As such, once you have sufficient size, it will more or less manage itself.

That isn't going to be an acceptable answer for a lot of the people that want to have more control over that, I know. There are two answers to that, the first is from the human perspective, the second is from enforcement perspective. Let us deal with the first one first.

Make it easy to do the right thing. I seem to be saying that a lot lately, but this is a core concept that all developers needs to assimilate.

If loading a user from the database is as simple as:

usersRepository.Load(userID);

Then you can be certain that no one will try to write this:

using(SqlConnection connection = new SqlConnection("Data Source=LocalHost;Initial Catalog=MyAppDB;"))

{

    connection.Open();
    SqlCommand cmd = connection.CreateCommand();
    cmd.CmmandText = "SELECT * FROM Users WHERE Id = " +userId;

}

Yes, it is that simple. And yes, this assume non malicious people in the team. If you have those, termination is recommended. It is your choice whatever it would be of them or their employment.

Enforcement is easy as well. Get NDepends and setup the appropriate CQL queries. Something like: error if class in "MyApp.UI" references "MyApp.Infrastructure".

The two project solution

Back to technical content :-)

This post from Kyle Baley got me thinking about my last few projects. In them, I trended toward the assembly per logical layer of the app. So I had things like:

  • MyApp.Model
  • MyApp.Web
  • MyApp.Services
  • MyApp.Infrastructure
  • MyApp.Tests

It worked, but at one point I found myself managing 18 projects solutions, and that was only because I was committed to reducing the number (otherwise it would have been much higher). I hear about people talking about 30 - 60 projects in a single solution, and there are people with more. Considering the cost of just managing that (not to mention the cost for compilation per project), that sounds like a very bad approach.

Right now, I think that on the next project I'll have the following structure:

  • MyApp
    • /Model
    • /Web
    • /Services
    • /Infrastructure
  • MyApp.Tests

Everything that the application does will be in a single project, without the need to split it off to a lot of separate projects. I found that the vaunted isolation that we seem to seek is achievable easily without forcing artificial barriers between the different parts of the application. It also means shorter compilation times and easier deployment mode.

More than everything else, it means that I have less noise to deal with.

I would extend that to an assembly per physical location, so assuming that I had a smart client application, I would use the following scheme:

  • MyApp.Server
  • MyApp.Interfaces
  • MyApp.Client
  • MyApp.Tests

And that would be it.

Thoughts?

On Security - From the physical perspective

image
                         (Image from Boing Boing)

I don't generally comment on such things, but in my recent trip to London, I had to go past the Heathrow Airport, and I found the security measures there... excessive. Especially in light of several other... observations that I made while in London. Those observations match across several countries (Canada, UK, Denmark, and I am pretty sure that it will match what I will find in USA). This is not an attack on the UK, this post was merely sparked by my recent trip there.

You could say that I have a professional interest in terror and terrorists. And I spent some time trying to understand them. Living in Israel, a country with a long and bloody history of dealing with terrorism, certainly help bring the lesson home, even if I haven't had personal experiences with it.

Israel's security forces, by and large, have long institutional memory with regard to how to handle potential threats, and I did my share of that a few times. As such, I have to say that while the security on Heathrow was excellent after you passed it, there seem to be zero security whatsoever until that point.

A casual survey of the way that I would attempt to make any sabotage attempt suggested several routes that seems very opened for attack.

Now, to be clear, I despise terrorism to the depth of my soul, and I have little but contempt for the actions of people who believe that random killings and the attacks on civilians are a viable method of operation.

I can think of at least three non violent ways to cause problems that are going to bring as much news coverage, cause no loss of life or damage to property and are even likely to be legal.

Oh, and they would be aggravating as hell itself.

That disclaimer aside, assuming that you can get a bomb (and you can find enough instructions in Google, even if you can't get a basic chemistry book), there is a lot of damage that can be done easily enough.

Just take the train. I got into the train by walking into the station with a large suitcase. At no point did I even see security, much less had them inspect what I was doing. I left the suitcase at one end of the train car and found an empty seat in almost the other end of the car.

During the one hour ride, it appears that no one has noticed that there was an unattended suitcase that was just sitting there for a long period.

Compare that to the experience that I expect to have in Israel. In nearly all public buildings (from the Mall to the Library), you have a guard that will ask to open your bags, and I can't imagine an unattended bag sitting anywhere with people for a long period of time. People here notice that (and trains, bus stations and even on the buses themselves there are security guys that will generally catch this type of thing).

Just to give you an example, here is what happens if you leave your laptop unattended in Israel. They shot it dead.

Israel has very good reasons for this type of behavior. I went for a pub with a friend from the states not long ago, and we sat down in a place on the shores of Tel Aviv, and talked a bit. The discussion moved to terror at some point, and I began to list bombings in the area. That very pub we were sitting on, a club not 500 meters away, a bus on the next street, etc. And that was what I could recall off hand and at a moment's notice.

I do not expect it to be the same elsewhere, but I was disturb by the difference between level of security for getting on a plane vs. everything else.

This post is getting long, morose and I have another flight to catch soon. It is also a subject that I tend not to talk about much, at least not on the blog, so I'll wrap it up now. But that dissonance is annoying as hell, disturbing at many levels and irritating just about all my professional senses.

Rose - Because I don't have any cat pictures to post

Well, apparently is is customary for bloggers that do not have anything to post to put some pictures of their cat. I don't have a cat, but I have a this shy little dog that I adore, her name is Rose.

image

Of course, that doesn't really give you any sense of scale, so here is her standing up, I am a big guy, her nose topped six feet by a fair margin. (about 1.85 for the civilized among us).

image

And, of course, the other side, here is her after I let her know that I am going away for a few weeks. I hope the bruises will heal soon, but for now, I need to move a bit carefully.

image

And no, it is not going to become a theme in this blog. But I found myself explaining to people on the phone all too often why they suddenly hear cries of "No, you are not supposed to eat cats!" and "Damn it already, I am not edible, get that through your head" or just "Argh!".

I hope this clarify things a bit.

Re: Versioning Issues With Abstract Base Classes and Interfaces

Phil Haack is talking about why the MS MVC team changed IHttpContext to HttpContextBase. I follow the argument, but at some point, I just lost it. This, in particular, had me scratching my head in confusion:

Adding this method doesn’t break older clients. Newer clients who might need to call this method can recompile and now call this new method if they wish. This is where we get the versioning benefits.

How on earth does adding a new method to an interface would break an existing client? How on earth does adding a new method to an interface require a compilation from client code. Now, to be clear, when I am talking about changing the interface I am talking solely about adding new methods (or overloads), not about changing a method signature or removing a method. Those are hard breaking changes no matter what approach you take. By adding a method to an interface? That is mostly harmless.

The only thing that require that is if you are implementing this interface, not if you are using it. It is possible that Phil is using the term clients in a far wider meaning than I would put on this, but this is not the general use of the term as I see it.

Prefer abstract classes to interfaces, use internals and no virtual by defaults are three of the main issues that I have with the framework design guidelines. I have major issues with them because they are actively harming the users of the framework. Sure, it might make my work as a framework developer harder, but guess what, that is my job as a framework developer, to give the best experience that I can to my users. [Framework Design Guidelines rant over]

Now, I do believe that I have some little experience in shipping large frameworks and versioning them through multiply releases and through a long period of time. I also believe that some of those frameworks are significantly larger and more complex than what the MS MVC is going to be. (Hint, if MS MVC even seems, by an illiterate drunken sailor on a rainy day, to approach NHibernate's complexity, it is time to hit the drawing board again).

And those frameworks are using frameworks to do pretty much everything. And I cannot recall off hand any breaking change that resulted from that. In some cases, where the interface is an extension point into the framework, we have gone with interface + base class with default functionality. If you use the base class, you are guaranteed no breaking change. The reasoning for an interface is that it is giving the user more choice, you aren't limiting the options that the user have when it comes to use this (by taking out the single inheritance slot, for example).

Now, if we analyze the expected usage of IHttpContext for a moment, who is going to be affected by changing this interface? Only implementers. Who is going to implement IHttpHandler? I can think of only two scenarios. Hand rolled test fakes and extending the Http Context in some interesting ways, perhaps by adding proxies or decorators to it.

In the first case, that is no something that I would worry about. The second is far rarer but also the much more interesting case, but those are generally not done by hand (I wouldn't want to type all the methods of IHttpContext, that is for sure). Even if it was, I still have no issue with it. New framework version, add a method. It is not a significant change. A significant change would require me to rework large swathes of my application.

Now, why do I care for that?

The reason is very simple. It is a pain to me, personally, when I end up running into those warts. It is annoying, frustrating and aggravating. I like to be happy, because otherwise I am not happy, so I try to minimize my exposure to the afore mentioned warts. Hopefully, I can make them go away entirely. And not just by pulling the blanket over my head.

Planning Rhino Mock 3.5 - The Lambda Edition

Note, this is a copy of the discussion that is taking place in the Rhino Mocks mailing list.

Now that I pushed all the current changes out, we need to start thinking about the next version. I am planning on seeing what kind of syntax I can get from the language using C# 3.0. As of yet, I don't have any concrete ideas about what should be done, so I am here to solicit ideas about what you want.

Syntax suggestions, patches, comments, etc are all welcome.

Rhino Mocks 3.4 Released

Well, it is about that time again, I got enough changes to make a release worthwhile. It has been almost five months, and we got a whole bunch of stuff going on, including a lot of stuff from contributors. (Take that, one man project :-) ).

As usual, you can get it here and the source is available here.

It is interesting to note that a lot of people have stuck with Rhino Mocks 3.1, I guess that there is really no need to upgrade, if you get what you want...

Features, more features, glorious features!

  • Added CreateMockWithRemoting and DynamicMockWithRemoting to work around BCL bug when overriding a generic method with generic constraint.
    This is useful when you want to force Rhino Mocks to use Remoting proxies (usually it just guess and you don't deal with this). That is needed because at certain cases ( generic methods with generic constraints ) there is a bug in Reflection Emit that generate invalid code.
  • Mike Nichols  has added List.ContainsAll() constraint
  • Shawn Neal added the DoNotExpect syntax, which is clearer the Expect.Call().Repeat.Never(), it is certainly shorter.
  • Added trace writer with stack trace, should make it easier to track down strange mocking behavior. Now you can just enable logging and see all the method calls that are being made.
  • Guido van Loon supplied the Property.AllPropertiesMatch constraints
  • Making sure that recorders that wrap recorders will be merged if they contain only one element. Again, this is something that you are not supposed to understand unless you went deep into the code. Suffice to say that the behavior is easier to understand now.

Bugs, annoying bugs, stupid bugs!

  • Fixing issue with empty recorders nested in recorders that has expectations. If you don't get that, then rest assure, neither do I.
  • Demid Sukhovsky  fixed Rhino Mocks support for Out/Ref values using remoting proxies.
  • Fixing null reference exception when stubbing a class with protected setter.
  • Aaron Jensen fixed the validation message when you expect a method that accept a mock object as a parameter.
  • Aaron Jensen also fixed a bunch of my typos.
  • Steven Rockarts fixed a bunch more of my typos.
  • Fixed an issue with methods returning non primitive value types, such as UInt32

Building Domain Specific Languages in Boo: Available for early access

image Well, this is a big moment. My book is now available for as part of the Manning Early Access Program.

Chapter 1 to 4 are already there, but they are there in their nearly raw format. That means that they have not been edited yet, and I didn't put in the results from the review cycle that we did yet.

The book forum is here, I eagerly awaits the posts telling me how off base I am :-)

More seriously, I would appreciate any feedback that you have on the book, the topic, the code, etc.

Thanks,

  ~Ayende

I think my brain just fried

I am working on ICriterion and IProjection at the moment, making each understand each other. It is working very well, and I really like the ability to just go in and make a drastic change.

The problem is the scope of the change. I have had to touch pretty much all of the NHibernate.Expression in the last two days. Just to give you an idea, here is the most recent comment.

Damn I am glad that I have tests there.

image

Linq for NHibernate Adventures

So for the last few hours I have been getting back into the Linq for NHibernate project, after having left it for far too long. I am beginning to think that building a Linq provider might not have been the best way to learn C# 3.0, but never mind that now.

It is with great pride that I can tell you that the following query now works:

from e in db.Employees
select new { Name = e.FirstName + " " + e.LastName, Phone = e.HomePhone }

Why is this trivial query important? Well, it is important because about a year ago I said that there is only a single feature in Linq for SQL that NHibernate doesn't have. I also promised to fix it the moment that someone would tell me what it is. No one did, so it waited until today.

The main issue had to do with the Criteria API and handling parameters, no one ever needed to do that, it seems. When they did, they generally used HQL, which did have this feature. Since I have based Linq for NHibernate on the Criteria API*, that was a problem.

Now that ReSharper works on C# 3.0, I can actually get things done there, so I sat down and implemented it. It is surprisingly difficult issue, but I threw enough code on it until it gave up (I wonder if there is a name for nested visitors... ).

At any rate, I strongly recommend that you'll take a look at the project. And bug (fixes) and other patches are welcome**.

* This decision was important for two reasons, the first was that it is generally easier to use the Criteria API programmatically, and the second was that I wanted to ensure that the Crtieria API (which I favor) was a full featured as the HQL route.

** Yes, I know that I'll regret it.

Amazon S3 Outage

imageAmazon has released (scroll all the way down) some details about their outage. It is light on details, but it gives enough information to guess a few things.

Apparently, there is an authentication services that handles both account validation and authentication of request. Since authenticated requests requires some cryptography, they are significantly more expensive than unauthenticated requests. Since the authentication services also handle account validation, and since that is done per request, swamping the authentication services with costly authenticated request has overloaded the capacity of the authentication services and cause many account validation requests to fail.

As I was reading Amazon's response, I kept thinking about Release It! It seems like Amazon's issue could be a great sample for that book.

I am not sure what could be done to prevent this rolling outage scenario (except adding more capacity, of course). Maybe prioritizations of requests, or determining that the authentication service is effectively down, triggering a Circuit Breaker and ignoring account validation for a specified amount of time?

That would give some people the option to use the service even if they didn't pay for the account, I would guess, but that would be much better for Amazon than the effect of the outage.

It is an interesting thought experiment.

What would you do in this scenario?

Random Musings

Okay, this is just a train of thought about a few things. I just finished submitting a couple of chapters, and I feel very wrung out at the moment. For the last few weeks I have been too busy to do much of the things that I usually do. First there was getting ready to quit my job, with a last minute project to finish. That took most of my time in that regard, and I let the book project slid too far behind schedule.

This week I spent most of my time writing, and it was bloody well hard. The chapter about integrating DSL into your application progressed agonizingly slowly. I couldn't really keep writing for extended period of time, which was damn annoying, since I already had a fairly good idea what I wanted to write. The next chapter was far easier to write, but still took too long, in my opinion.

One of the results of that was that I didn't pay much attention to email, currently I have 64(!!!) unread emails, and about a dozen tasks that I need to handle relating to my OSS projects. I thought leaving work was supposed to give me more free time... :-) I am going to get some rest and try to handle that afterward.

Other tidbits include the fact that ReSharper 4.0 is now in early access program, with features that really make me want to drool. This means that I can work with VS 2008 now (well, hopefully). This has direct correlation to the amount of work that is now going to head into Linq for NHibernate.

Udi Dahan has a very interesting post about moving from CRUD to Domain Driven services. It is an eye opener, and I heartily recommend reading it.

Amazon's S3 have suffered a massive failure, apparently. Some of the comments on that are interesting, looks like it is suggested as high volume, good performance but with possible high latencies. As such, you probably should use it for archiving and storage, but still keep a hot cache around on your own servers so you can get at least partial service when it is down.

Enough for now, I think...