Ayende @ Rahien

Code crimes, because even the Law needs to be broken

Mon, 11 Jun 2012 09:00:00 GMT

In 99.9% of the cases, if you see this, it is a mistake:

But I recently came into a place where this is actually a viable solution to a problem. The issue occurs deep inside RavenDB. And it boils down to a managed application not really having any way of actually assessing how much memory it is using.

What I would really like to be able to do is to specify a heap (using HeapCreate) and then say: “The following objects should be created on this heap”, which would allow to me to control exactly how much memory I am using for a particular task. In practice, that isn’t really possible (if you know how, please let me know).

What we do instead is to use as much memory as we can, based on the workload that we have. At certain point, we may hit an Out of Memory condition, which we take to mean, not that there is no more memory, but the system’s way of telling us to calm down.

By the time we reached the exception handler, we already lost all of the current workload, that means that now there is a LOT of garbage around for the GC. Before, when it tried to cleanup, we were actually holding to a lot of that memory, but not anymore.

Once an OOME happened, we can adjust our own behavior, to know that we are consuming too much memory and should be more conservative about our operations. We basically reset the clock back and become twice as conservative about using more memory. And if we get another OOME? We become twice as conservative again , and so on.

Eventually, even under high workloads and small amount of memory, we complete the task, and we can be sure that we make effective use of the system’s resources available to us.

Beware of big Task Parallel Library Operations

Thu, 12 Apr 2012 09:54:00 GMT

Take a look at the following code:

class Program
{
    static void Main()
    {
        var list = Enumerable.Range(0, 10 * 1000).ToList();

        var task = ProcessList(list, 0);


        Console.WriteLine(task.Result);

    }

    private static Task<int> ProcessList(List<int> list, int pos, int acc = 0)
    {
        if (pos >= list.Count)
        {
            var tcs = new TaskCompletionSource<int>();
            tcs.TrySetResult(acc);
            return tcs.Task;
        }

        return Task.Factory.StartNew(() => list[pos] + acc)
            .ContinueWith(task => ProcessList(list, pos + 1, task.Result))
            .Unwrap();
    }
}

This is a fairly standard piece of code, which does a “complex” async process and then move on. It is important in this case to do the operation in the order they were given, and the real code is actually doing something that need to be async (go and fetch some data from a remote server).

It is probably easier to figure out what is going on when you look at the C# 5.0 code:

class Program
{
    static void Main()
    {
        var list = Enumerable.Range(0, 10 * 1000).ToList();

        var task = ProcessList(list, 0);

        Console.WriteLine(task.Result);

    }

    private async static Task<int> ProcessList(List<int> list, int pos, int acc = 0)
    {
        if (pos >= list.Count)
        {
            return acc;
        }

        var result = await Task.Factory.StartNew(() => list[pos] + acc);

        return await ProcessList(list, pos + 1, result);
    }
}

I played with user mode scheduling in .NET a few times in the past, and one of the things that I was never able to resolve properly was the issue of the stack depth. I hoped that the TPL would resolve it, but it appears that it didn’t. Both code samples here will throw StackOverFlowException when run.

It sucks, quite frankly. I understand why this is done this way, but I am quite annoyed by this. I expected this to be solved somehow. Using C# 5.0, I know how to solve this:

class Program
{
    static void Main()
    {
        var list = Enumerable.Range(0, 10 * 1000).ToList();

        var task = ProcessList(list);

        Console.WriteLine(task.Result);

    }

    private async static Task<int> ProcessList(List<int> list)
    {
        var acc = 0;
        foreach (var i in list)
        {
            var currentAcc = acc;
            acc += await Task.Factory.StartNew(() => i + currentAcc);
        }
        return acc;
    }
}

The major problem is that I am not sure how to translate this code to C# 4.0. Any ideas?

Memorable code

Thu, 27 Oct 2011 10:00:00 GMT

public class Program
{
    static List<Thread> list = new List<Thread>();
    private static void Main(string[] args)
    {
        var lines = File.ReadAllLines(args[0]);

        foreach (var line in lines)
        {
            var t = new Thread(Upsert)
            {
                Priority = ThreadPriority.Highest,
                IsBackground = true
            };
            list.Add(t);
            t.Start(line);
        }

        foreach (var thread in list)
        {
            thread.Join();
        }

    }

    private static void Upsert(object o)
    {
        var args = o.ToString().Split(',');
        try
        {
            using(var con = new SqlConnection(Environment.CommandLine.Split(' ')[1]))
            {
                var cmd = new SqlCommand
                {
                    Connection = con, 
                    CommandText = "INSERT INTO Accounts VALUES(@p1, @p2, @p3, @p4,@p5)"
                };

                for (var index = 0; index < args.Length; index++)
                {
                    cmd.Parameters.AddWithValue(@"@p" + (index + 1), args[index]);
                }

                try
                {
                    cmd.ExecuteNonQuery();
                }
                catch (SqlException e)
                {
                    if(e.Number == 2627 )
                    {
                        cmd.CommandText = "UPDATE Accounts SET Name = @p2, Email = @p3, Active = @p4, Birthday = @p5 WHERE ID = @p1";
                        cmd.ExecuteNonQuery();
                    }
                }
            }
        }
        catch (SqlException e)
        {
            if(e.Number == 1205)
            {
                var t = new Thread(Upsert)
                {
                    Priority = ThreadPriority.Highest,
                    IsBackground = true
                };
                list.Add(t);
                t.Start(o);
            }
        }
    }
}

Negative hiring decisions, Part II

Mon, 10 Oct 2011 10:00:00 GMT

Another case of a candidate completing a task at home and sending it to me which resulted in a negative hiring decision is this:

protected void Button1_Click(object sender, EventArgs e)
{
    string connectionString = @"Data Source=OFFICE7-PC\SQLEXPRESS;Integrated Security=True";
    string sqlQuery = "Select UserName From  [Users].[dbo].[UsersInfo] Where UserName = ' " + TextBox1.Text + "' and Password = ' " + TextBox2.Text+"'";
    
    using (SqlConnection connection = new SqlConnection(connectionString))
    {
        SqlCommand command = new SqlCommand(sqlQuery, connection);
        connection.Open();
        SqlDataReader reader = command.ExecuteReader();
        try
        {
            while (reader.Read())
            {
           
            }
        }
        finally
        {
            if (reader.HasRows)
            {
                reader.Close();
                Response.Redirect(string.Format("WebForm2.aspx?UserName={0}&Password={1}", TextBox1.Text, TextBox2.Text));
            }
            else
            {
                reader.Close();
                Label1.Text = "Wrong user or password";
            }
        }
    }
}

The straw that really broke the camel’s back in this case was the naming of WebForm2. I could sort of figure out the rest, but not bothering to give a real name to the page was over the top.

Is Node.cs a cure for cancer?

Wed, 05 Oct 2011 10:00:00 GMT

This is mainly a tongue in cheek post, in reply to this guy. I decided to take his scenario and try it using my Node.cs “framework”. Here is the code:

public class Fibonaci : AbstractAsyncHandler
{
    protected override Task ProcessRequestAsync(HttpContext context)
    {
        return Task.Factory.StartNew(() =>
        {
            context.Response.ContentType = "text/plain";
            context.Response.Write(Fibonacci(40).ToString());
        });
    }

    private static int Fibonacci(int n)
    {
        if (n < 2)
            return 1;
        
        return Fibonacci(n - 2) + Fibonacci(n - 1);
    }
}

We start by just measuring how long it takes to serve a single request:

$ time curl http://localhost/Fibonaci.ashx
165580141
real    0m2.763s
user    0m0.000s
sys     0m0.031s

That is 2.7 seconds for a highly compute bound operation. Now, let us see what happens when we use Apache Benchmark to test things a little further:

ab.exe -n 10 -c 5 http://localhost/Fibonaci.ashx

(Make a total of ten requests, maximum of 5 concurrent ones)

And this gives us:

Requests per second:    0.91 [#/sec] (mean)
Time per request:       5502.314 [ms] (mean)
Time per request:       1100.463 [ms] (mean, across all concurrent requests)

Not bad, considering the best node.js (on a different machine and hardware configuration) was able to do was 0.17 requests per second.

Just for fun, I decided to try it with a hundred requests, with 25 of them concurrent.

Requests per second:    0.97 [#/sec] (mean)
Time per request:       25901.481 [ms] (mean)
Time per request:       1036.059 [ms] (mean, across all concurrent requests)

Not bad at all.

What is the cost of try/catch

Wed, 14 Sep 2011 09:00:00 GMT

I recently got a question about the cost of try/catch, and whatever it was prohibitive enough to make you want to avoid using it.

That caused some head scratching on my part, until I got the following reply:

But, I’m still confused about the try/catch block not generating an overhead on the server.
Are you sure about it?

I learned that the try block pre-executes the code, and that’s why it causes a processing overhead.
Take a look here: http://msdn.microsoft.com/en-us/library/ms973839.aspx#dotnetperftips_topic2

Maybe there is something that I don’t know? It is always possible, so I went and checked and found this piece:

Finding and designing away exception-heavy code can result in a decent perf win. Bear in mind that this has nothing to do with try/catch blocks: you only incur the cost when the actual exception is thrown. You can use as many try/catch blocks as you want. Using exceptions gratuitously is where you lose performance. For example, you should stay away from things like using exceptions for control flow.

Note that the emphasis is in the original. There is no cost to try/catch the only cost is when an exception is thrown, and that is regardless of whatever there is a try/catch around it or not.

Here is the proof:

var startNew = Stopwatch.StartNew();
var mightBePi = Enumerable.Range(0, 100000000).Aggregate(0d, (tot, next) => tot + Math.Pow(-1d, next)/(2*next + 1)*4);
Console.WriteLine(startNew.ElapsedMilliseconds);

Which results in: 6015 ms of execution.

Wrapping the code in a try/catch resulted in:

var startNew = Stopwatch.StartNew();
double mightBePi = Double.NaN;
try
{
    mightBePi = Enumerable.Range(0, 100000000).Aggregate(0d, (tot, next) => tot + Math.Pow(-1d, next)/(2*next + 1)*4);
}
catch (Exception e)
{
    Console.WriteLine(e);
}
Console.WriteLine(startNew.ElapsedMilliseconds);

And that run in 5999 ms.

Please note that the perf difference is pretty much meaningless (only 0.26% difference) and is well within the range of derivations for tests runs.

A surprise TaskCancelledException

Fri, 09 Sep 2011 09:00:00 GMT

All of a sudden, my code started getting a lot of TaskCancelledException. It took me a while to figure out what was going on. We can imagine that the code looked like this:

var unwrap = Task.Factory.StartNew(() =>
{
    if (DateTime.Now.Month % 2 != 0)
        return null;

    return Task.Factory.StartNew(() => Console.WriteLine("Test"));
}).Unwrap();

unwrap.Wait();

The key here is that when Unwrap is getting a null task, it will throw a TaskCancelledException, which was utterly confusing to me. It make sense, because if the task is null there isn’t anything that the Unwrap method can do about it. Although I do wish it would throw something like ArgumentNullException with a better error message.

The correct way to write this code is to have:

var unwrap = Task.Factory.StartNew(() =>
{
    if (DateTime.Now.Month % 2 != 0)
    {
        var taskCompletionSource = new TaskCompletionSource<object>();
        taskCompletionSource.SetResult(null);
        return taskCompletionSource.Task;
    }

    return Task.Factory.StartNew(() => Console.WriteLine("Test"));
}).Unwrap();

unwrap.Wait();

Although I do wish that there was an easier way to create a completed task.

Concurrent Max

Sun, 12 Jun 2011 05:43:00 GMT

Can you think of a better way to implement this code?

private volatile Guid lastEtag;
private readonly object lastEtagLocker = new object();
internal void UpdateLastWrittenEtag(Guid? etag)
{
    if (etag == null)
        return;

    var newEtag = etag.Value.ToByteArray();

    // not the most recent etag
    if (Buffers.Compare(lastEtag.ToByteArray(), newEtag) <= 0)
    {
        return;
    }

    lock (lastEtagLocker)
    {
        // not the most recent etag
        if (Buffers.Compare(lastEtag.ToByteArray(), newEtag) <= 0)
        {
            return;
        }

        lastEtag = etag.Value;
    }
}

We have multiple threads calling this function, and we need to ensure that lastEtag value is always the maximum value. This has the potential to be called often, so I want to make sure that I chose the best way to do this. Ideas?

ConcurrentDIctionary.GetOrAdd may call the valueFactory method more than once

Tue, 05 Apr 2011 09:00:00 GMT

Originally posted at 3/28/2011

When you assume, you are making an ass out of yourself, you stupid moronic idiot with no sense whatsoever. The ass in question, if anyone cares to think about it, is yours truly.

Let us take a look at the following code snippet:

var concurentDictionary = new ConcurrentDictionary<int, int>();
var w = new ManualResetEvent(false);
int timedCalled = 0;
var threads = new List<Thread>();
for (int i = 0; i < Environment.ProcessorCount; i++)
{
    threads.Add(new Thread(() =>
    {
        w.WaitOne();
        concurentDictionary.GetOrAdd(1, i1 =>
        {
            Interlocked.Increment(ref timedCalled);
            return 1;
        });
    }));
    threads.Last().Start();
}

w.Set();//release all threads to start at the same time
foreach (var thread in threads)
{
    thread.Join();
}

Console.WriteLine(timedCalled);

What would you say would be the output of this code?

Well, I assumes that it would behave in an atomic fashion, that the implementation is something like:

if(TryGetValue(key, out value))
   return value;

lock(this)
{
   if(TryGetValue(key, out value))
      return value;

   AddValue( key, valueFactory());
}

Of course, the whole point of the ConcurentDictionary is that there are no locks. Well, that is nice, except that because I assumed that the call is only made once, I called that with a function that had side effects when called twice.

That was a pure hell to figure out, because in my mind, of course that there was no error with this function.

Synchronization primitives, MulticastAutoResetEvent

Tue, 26 Oct 2010 10:00:00 GMT

I have a very interesting problem within RavenDB. I have a set of worker processes that all work on top of the same storage. Whenever a change happen in the storage, they wake up and start working on it. The problem is that this change may be happening while the worker process is busy doing something other than waiting for work, which means that using Monitor.PulseAll, which is what I was using, isn’t going to work.

AutoResetEvent is what you are supposed to use in order to avoid losing updates on the lock, but in my scenario, I don’t have a single worker, but a set of workers. And I really wanted to be able to use PulseAll to release all of them at once. I started looking at holding arrays of AutoResetEvents, keeping tracking of all changes in memory, etc. But none of it really made sense to me.

After thinking about it for a while, I realized that we are actually looking at a problem of state. And we can solve that by having the client hold the state. This led me to write something like this:

public class MultiCastAutoResetEvent 
{
    private readonly object waitForWork = new object();
    private int workCounter = 0;
    
    
    public void NotifyAboutWork()
    {
        Interlocked.Increment(ref workCounter);
        lock (waitForWork)
        {
            Monitor.PulseAll(waitForWork);
            Interlocked.Increment(ref workCounter);
        }
    }
    
    
    public void WaitForWork(TimeSpan timeout, ref int workerWorkCounter)
    {
        var currentWorkCounter = Thread.VolatileRead(ref workCounter);
        if (currentWorkCounter != workerWorkCounter)
        {
            workerWorkCounter = currentWorkCounter;
            return;
        }
        lock (waitForWork)
        {
            currentWorkCounter = Thread.VolatileRead(ref workCounter);
            if (currentWorkCounter != workerWorkCounter)
            {
                workerWorkCounter = currentWorkCounter;
                return;
            }
            Monitor.Wait(waitForWork, timeout);
        }
    }
}

By forcing the client to pass us the most recently visible state, we can efficiently tell whatever they still have work to do or do they have to wait.

JAOO: The Go Programming Language

Wed, 06 Oct 2010 10:00:00 GMT

Currently sitting in the Go language keynote in JAOO.

Go was created to handle Google needs:

Compilation speed
Efficient
Fit for the current set of challenges we face

Go is meant to be Simple, Orthogonal, Succinct. Use C syntax, but leave a lot of complexity behind.

This make sense, because C/C++ is horrible for a lot of reasons, because of complexity added to it over the years. Actually, the major problem is that C/C++ mindset are literally designed for a different era. Take into account how long it can take to build a C++ application (I have seen no too big apps that had half an hour build times!), and you can figure out why Google wanted to have a better model.

Nice quotes:

Programming shouldn’t be a game of Simon’s Says
Difference between seat belts & training wheels – I am going to use that a lot.

I find Go’s types annoying:

[variable name] [variable type] vs. [variable type] [variable name]
And []int vs. int[]

I like the fact that is uses garage collection in native code, especially since if you reference a variable (including local) it will live as long as it has a reference to. Which is a common mistake in C.

It also have the notion of deferring code. This is similar to how you can use RAII in C++, but much more explicit, which is good, because RAII is always tricky. It very cleanly deals with the need to dispose things.

All methods looks like extension methods. Data is held in structures, not in classes. There is support for embedding structures inside one another, but Go isn’t an OO language.

Interfaces work for all types, and interfaces are satisfied implicitly! Much easier to work, think about this like super dynamic interfaces, but strong typed. Implies that we can retrofit things afterward.

Error handling is important, and one of the major reasons that I moved from C++ to .NET. Go have two modes, the first is error codes, which they use for most errors. This is especially nice since Go have multiple return values, so it is very simple to use. But it also have the notion of panics, which are similar to exceptions. Let us take a look at the following test code:

package main

func test() {
  panic("ayende")
}

func main() {
    test()
}

Which generate the following panic:

panic: ayende

panic PC=0x2ab34dd47040
runtime.panic+0xb2 /sandbox/go/src/pkg/runtime/proc.c:1020
    runtime.panic(0x2ab300000000, 0x2ab34dd470a0)
main.test+0x47 /tmp/gosandbox-a9aaff6c_68e6b411_26fb5255_aa397bb1_6299a954/prog.go:5
    main.test()
main.main+0x18 /tmp/gosandbox-a9aaff6c_68e6b411_26fb5255_aa397bb1_6299a954/prog.go:8
    main.main()
mainstart+0xf /sandbox/go/src/pkg/runtime/amd64/asm.s:78
    mainstart()
goexit /sandbox/go/src/pkg/runtime/proc.c:145
    goexit()

I got my stack trace, so everything is good. Error handling is more tricky. There isn’t a notion of try/catch, because panics aren’t really exceptions, but you can recover from panics:

package main
import ( 
   "fmt"
)
func badCall() {
  panic("ayende")
}

func test() {
   defer func() { 
    if e := recover(); e != nil {
       fmt.Printf("Panicing %s\r\n", e);
    }
   
    }()
    badCall()
   fmt.Printf("After bad call\r\n");
}

func main() {
   fmt.Printf("Calling test\r\n");
   test()
   fmt.Printf("Test completed\r\n");
 
 
}

Which will result in the following output:

Calling test
Panicing ayende
Test completed

All in all, if I need to write native code, I’ll probably go with Go instead of C now. Especially since the ideas that it have about multi threading are so compelling. I just wish that the windows port would be completed soon.

It ain’t no simple feature, mister

Tue, 14 Sep 2010 10:00:00 GMT

I recently got a bug report about NH Prof in a multi monitor environment. Now, I know that NH Prof works well in multi monitor environment, because I frequently run in such an environment myself.

The problem turned out to be not multi monitors in and of itself, but rather how NH Prof handles the removal of a monitor. It turns out that NH Prof has a nice little feature that actually remembers the last position the window was at, and returns to it on start. When the monitor NH Prof was located on was removed, on start NH Prof would put itself beyond the screen position.

That led to me having to figure out how to find the available monitor space, so I could detect if the saved positions were valid or not. What I found interesting in this is that what seemed to be a very trivial feature (save two numbers) turned out to be somewhat more complex, and I am pretty sure that there are other scenarios that I am missing (in the very same feature).

Implementing CreateSequentialUuid()

Mon, 13 Sep 2010 10:00:00 GMT

We run into an annoying problem in Raven regarding the generation of sequential guids. Those are used internally to represent the etag of a document.

For a while, we used the Win32 method CreateSequentialUuid() to generate that. But we run into a severe issue with that, it create sequential guids only as long as the machine is up. After a reboot, the guids are no longer sequential. That is bad, but it also means that two systems calling this API can get drastically different results (duh! that is the point, pretty much, isn’t it?). Which wouldn’t bother me, except that we use etags to calculate the freshness of an index, so we have to have an always incrementing number.

How would you implement this method?

public static Guid CreateSequentialUuid()

A few things to note:

We really actually care about uniqueness here, but only inside a single process, not globally.

The results must always be incrementing.

The always incrementing must be consistent across machine restarts and between different machines.

Yes, I am fully aware of the NHibernate’s implementation of guid.comb that creates sequential guids. It isn't applicable here, since it doesn't create truly sequential guids, only guids that sort near one another.

LightSwitch: Initial thoughts

Wed, 25 Aug 2010 04:52:00 GMT

As promised, I intend to spend some time today with LightSwitch, and see how it works. Expect a series of post on the topic. In order to make this a read scenario, I decided that that a simple app recording animals and their feed schedule is appropriately simple.

I created the following table:

Note that it has a calculated field, which is computed using:

There are several things to note here:

ReSharper doesn’t work with LightSwitch, which is a big minus to me.
The decision to use partial methods had resulted in really ugly code.
Why is the class called Animals? I would expect to find an inflector at work here.
Yes, the actual calculation is crap, I know.

This error kept appearing at random:

It appears to be a known issue, but it is incredibly annoying.

This is actually really interesting:

You can’t really work with the app unless you are running in debug mode. That isn’t the way I usually work, so it is a bit annoying.
More importantly, it confirms that this is indeed KittyHawk, which was a secret project in 2008 MVP Summit that had some hilarious aspects.

There is something that is really interesting, it takes roughly 5 – 10 seconds to start a LS application. That is a huge amount of time. I am guessing, but I would say that a lot of that is because the entire UI is built dynamically from the data source.

That would be problematic, but acceptable, except that it takes seconds to load data even after the app has been running for a while. For example, take a look here:

This is running on a quad core, 8 GB machine, in 2 tiers mode. It takes about 1 – 2 seconds to load each screen. I was actually able to capture a screen half way loaded. Yes, it is beta, I know. Yes, perf probably isn’t a priority yet, but that is still worrying.

Another issue is that while Visual Studio is very slow, busy about 50% of the time. This is when the LS app is running or not. As an a side issue, it is hard to know if the problem is with LS or VS, because of all the problems that VS has normally.

As an example of that, this is me trying to open the UserCode, it took about 10 seconds to do so.

What I like about LS is that getting to a working CRUD sample is very quick. But the problems there are pretty big, even at a cursory examination. More detailed posts touching each topic are coming shortly.

Runtime code compilation & collectible assemblies are no go

Tue, 24 Aug 2010 09:00:00 GMT

The problem is quite simple, I want to be able to support certain operation on Raven. In order to support those operations, the user need to be able to submit a linq query to the server. In order to allow this, we need to accept a string, compile it and run it.

So far, it is pretty simple. The problem begins when you consider that assemblies can’t be unloaded. I was very hopeful when I learned about collectible assemblies in .NET 4.0, but they focus exclusively on assemblies generated from System.Reflection.Emit, while my scenario is compiling code on the fly (so I invoke the C# compiler to generate an assembly, then use that).

Collectible assemblies doesn’t help in this case. Maybe, in C# 5.0, the compiler will use SRE, which will help, but I don’t hold much hope there. I also checked out Mono.CSharp assembly, hoping that maybe it can do what I wanted it to do, but that suffer from the memory leak as well.

So I turned to the one solution that I knew would work, generating those assemblies in another app domain, and unloading that when it became too full. I kept thinking that I can’t do that because of the slowdown with cross app domain communication, but then I figured that I am violating one of the first rules of performance: You don’t know until you measure it. So I set out to test it.

I am only interested in testing the speed of cross app domain communication, not anything else, so here is my test case:

public class RemoteTransformer : MarshalByRefObject
{
    private readonly Transformer transfomer = new Transformer();

    public JObject Transform(JObject o)
    {
        return transfomer.Transform(o);
    }
}

public class Transformer
{
    public JObject Transform(JObject o)
    {
        o["Modified"] = new JValue(true);
        return o;
    }
}

Running things in the same app domain (base line):

static void Main(string[] args)
{
    var t = new RemoteTransformer();
    
    var startNew = Stopwatch.StartNew();

    for (int i = 0; i < 100000; i++)
    {
        var jobj = new JObject(new JProperty("Hello", "There"));

        t.Transform(jobj);

    }

    Console.WriteLine(startNew.ElapsedMilliseconds);
}

This consistently gives results under 200 ms (185ms, 196ms, etc). In other words, we are talking about over 500 operations per millisecond.

What happen when we do this over AppDomain boundary? The first problem I run into was that the Json objects were serializable, but that was easy to fix. Here is the code:

 static void Main(string[] args)
 {
    var appDomain = AppDomain.CreateDomain("remote");
    var t = (RemoteTransformer)appDomain.CreateInstanceAndUnwrap(typeof(RemoteTransformer).Assembly.FullName, typeof(RemoteTransformer).FullName);
    
    var startNew = Stopwatch.StartNew();
     
     for (int i = 0; i < 100000; i++)
     {
         var jobj = new JObject(new JProperty("Hello", "There"));

         t.Transform(jobj);

     }

     Console.WriteLine(startNew.ElapsedMilliseconds);
 }

And that run close to 8 seconds, (7871 ms). Or over 40 times slower, or just about 12 operations per millisecond.

To give you some indication about the timing, this means that an operation over 1 million documents would spend about 1.3 minutes just serializing data across app domains.

That is… long, but it might be acceptable, I need to think about this more.

LightSwitch: The Return Of The Secretary

Thu, 05 Aug 2010 10:00:00 GMT

Microsoft LightSwitch is a new 4GL tool from Microsoft, this is another in the series of “you don’t have to write any code” tools that I have seen.

Those are the tools that will give the secretary the ability to create applications and eliminate the need for coders. The industry has been chasing those tools since the 80s (does anyone remember the promises of the CASE tools?). We have seen many attempts at doing this, and all of them have run into a wall pretty quickly.

Oh, you can build a tool that gives you UI on top of a data store pretty easily. And you can go pretty far with it, but eventually your ability to point & click hit the limit, and you have to write code. And that is things totally breaks down.

LightSwitch is not yet publically available, so I have to rely on the presentation that Microsoft published. And I can tell you that I am filled with dread, based on what I have seen.

First of all, I strongly object to the following slide. Because I have the experience to know that working with a tool like that is akin to do back flips with a straightjacket on.

The capabilities of the tools that were shown in the presentation have strongly underwhelmed me in terms of newness, complexity or applicability.

Yeah, a meta data driven UI. Yeah, it can do validation on a phone number automatically (really, what happen with my Israeli based phone number?), etc. What is worse, even through the demo, I get the very strong feeling that the whole things is incredibly slow, you can see in the presentation multi second delays between screen repaints.

Then there are things like “it just works as a web app or windows app” which is another pipe dream that the industry has been chasing for a while. And the only piece of code that I have seen is this guy:

Which makes me want to break down and cry.

Do you know why? Because this is going to be the essence of a SELECT N+1 in any system, because this code is going to run once per each row in the grid. And when I can find bugs from watching a presentation, you know that there are going to be more issues.

So, just for fun sake, since I don’t have the bits and I can rely only on the presentation, I decided to make a list of all the things that are likely to be wrong with LightSwitch.

I’ll review it when it comes out, and if it does manage to do everything that it does and still be a tool usable by developers, I’ll have to eat crow (well, Raven :-) ), but I am not overly worried.

Here are a few areas where I am feeling certain things will not work right:

Source control – how do I diff two versions of the app to see what changes? Are all changes diffable?
Programmatic modifications:

what happen when I want to write some code to do custom validation of property (for instance, calling a web service)?
what happen when I want to put a custom control on the screen (for instance, a google maps widget)?

Upsizing – when it gets to a 1,000 users and we need a full blown app, how hard it is to do?
Performance – as I said, I think it is slow from the demo.
Data access behavior – from what I have seen so far, I am willing to be that it hits its data store pretty heavily.

I fully recognize that there is a need for such a tool, make no mistake. And giving users the ability to do that is important. What I strongly object to is the notion that it would be useful for developers writing real apps, not forms over data. To put it simply, simple forms over data is a solved problem. There is a large number of tools out there to do that. From Access to Oracle Apex to FoxPro. Hell, most CRM solutions will give you just that.

My concern is that there seems to be an emphasis on that being useful for developers as well, and I strongly doubt that.

Package management for .NET: Nu

Sun, 01 Aug 2010 09:00:00 GMT

Package management for .NET has long been an issue. I won’t rehash the story, but you can read about a lot of the problems here.

There has been several attempts to replicate the success of Gems in Ruby, but none has really taken hold. Dru Sellers & Rob Reynolds had decided to take a slightly different approach. Instead of trying to replicate Ruby’s gems, just use them.

That is how Nu was created (and I really want to know what that name was picked, I nearly titled this post Dr. No). Before you can use Nu, you need to install it, which means that you have to make the following one time steps:

Install Ruby (you can also use Iron Ruby, but the Ruby installer is just sweet & painless).
gem install nu

Once this is done, all you need to do to install a piece of software is just write:

nu install rhino.mocks
nu install nhibernate

And Nu will get the latest version (including resolving dependencies) and put them in your lib folder.

After executing the two commands above, I have the following contents in my lib directory.

castle.core
castle.dynamicproxy2
log4net
nhibernate
rhino.mocks

I like it, it is going to make things so much simpler!

A question of an untenable situation

Thu, 29 Jul 2010 09:00:00 GMT

One of the most common issues that I run into in my work is getting all sort of questions which sound really strange. For example, I recently got a question that went something like this:

What is the impact of reflection on NHibernate’s performance?

I started to answer that there isn’t one that you would notice, even beyond the major optimizations that NH had in that role, you are accessing a remote database, which is much more expensive. But then they told me that they profiled the application and found some stuff there about that.

I asked them what their scenario was, and the exchange went like that:

Well, when we load a million rows…

And that is your problem…

To be fair, they actually had a reasonable reason to want to do that. I disagree with the solution that they had, but it was a reasonable approach to the problem at hand.

Buy vs. Build & YAGNI

Fri, 16 Jul 2010 09:28:30 GMT

I was recently at the Israeli ALT.Net tools night, and I had a very interesting discussion on installers. Installers are usually a very painful part of the release procedure. The installer format for Windows is MSI, which is… strange. It takes time to understand how MSI work, and even after you got that, it is still painful to work with. Wix is a great improvement when it comes to building MSI installations, but that doesn’t make it good. Other installer builders, such as InstallSheild and NSIS are just as awkward.

The discussion that I had related to the complexity of building an installer on those technologies.

My argument was that it simply made no sense to try to overcome the hurdles of the installer technologies, instead, we can write our own installer more easily than fussing about the existing ones. The installer already assumes the presence of the .NET framework, so that make things even easier.

This is an application of a principle that I strongly believes in: Single purpose, specially built tools & components can be advantageous over more generic ones, for your specific scenarios.

Case in point, the installer. Installers are complicated beasts because they must support a lot of complex scenarios (upgrading from 5.3.2 to 6.2.1, for example), be transactional, support installation, etc. But for the installer in question, upgrade is always an uninstall of the previous version & install of the new one, and the only tasks it requires is copying files and modifying registry entries.

Given that set of requirements, we can design the following installer framework:

public interface IInstallerTask
{
     void Install();
     void Uninstall();
}

public class FileCopyTask : IInstallerTask
{
    public string Source { get;set; }
    public string Destination { get;set; }
    
    public void Install()
    {
        File.Copy(Source, Destination,overwrite:true);
    }
    
    public void Uninstall()
    {
        File.Delete(Destination);
    }
}

And building a particular installer would be:

ExecuteInstaller(
    Directory.GetFiles(extractedTempLocation)
        .Select( file =>
            new FileCopyTask
            {
                Source = file,
                Destination = Path.Combine(destinationPath, file)
            }        
        ),
    new RegistryKeyTask
    {
        Key = "HKLM/Windows/CurrentVersion...",
        Value = 9
    }
);

This gives the ExecuteInstaller method a list of tasks to be executed, which can then be used to installer or uninstall everything.

Yes, it is extremely simple, and yes, it wouldn’t fit many scenarios. But, it is quick to do, match the current and projected requirements, doesn’t introduce any new technology to the mix and it works.

Contrast that with having someone on the team that is the Installer expert (bad) or having to educate the entire team about installer (expensive).

The pitfalls of transparent security

Wed, 30 Jun 2010 07:42:00 GMT

A long time ago, I needed to implement a security subsystem for an application. I figured that it was best to make the entire security subsystem transparent to the developer, under the assumption that if the infrastructure would take care of security, it would make a lot more sense than relying on the developer to remember to add the security calls.

It took me a long while to realize how wrong that decision was. Security is certainly important, but security doesn’t apply to the system itself. In other words, while a specific user may not be allowed to read/write to the audit log, actions that the user made should be written to that log. That is probably the simplest case, but that are a lot of similar ones.

Ever since then, I favored using an explicit approach:

var books = session.CreateQuery("from Books")
                        .ThatUserIsAllowedToRead(CurrentUser)
                        .List<Book>();

This also help you implement more interesting features, such as “on behalf of”. And yes, it does put the onus of security on the developer, but considering the alternative, that is a plus.