Programming

Stop! Or I am writing another framework!

Ayende Rahien — Thu, 13 Nov 2008 03:40:38 GMT

There is a small dwarf in my head that whispers about persistent binary trees and RESTful services, views using DSL and JSON data interchange.

In other words, I really want to write NCouchDB. I had the exact same thing happening with queuing, and it didn't stop until I wrote Rhino Queues (five or six times!)

I keep trying to justify to myself that this is needed. But I already busy over my head. Please, help me stop this, before I lose all control.

Get that queue out of my head

Ayende Rahien — Sun, 21 Sep 2008 09:18:05 GMT

For some reason, I am having a lot of trouble lately getting Rhino Queues out of my head. I don't actually have a project to use it on, but it keep popping up in my mind. Before committing to the fifth or sixth rewrite of the project, I decided to take a stub at testing its performance.

The scenario is sending a hundred thousand (100,000) messages, with sizes ranging from 4Kb to 16Kb, which are the most common sizes for most messages.

On my local machine, I can push that amount of data in 02:02.02. So just above two minutes. This is for pushing 1,024,540,307 bytes, or just under a gigabyte in size. This translate to over 800 messages a second and close to 8.5 MB per second.

Considering that I am pushing everything over HTTP, that is just fine by me. But the real test would be to see how it works over the Internet.

I have a limited upload capacity here, so trying to push a gigabyte up would take too long. Instead, I confined myself to 5,000 messages, of the same varying sizes as before.

This time, over the Internet, it took 27:25:95 minutes. Just short of half an hour to push 51,135,538 bytes (~48Mb).

But that is testing throughput. While that is interesting on its own, it doesn't really tell us much about another important quality of the system, latency. Over the Internet, it took an average of 25 seconds to send a message batch. This is pretty fast, but I don't think it is great.

Anyway, getting back into the code didn't really help much, and I think that I will need to rewrite it again.

What I am working on...

Ayende Rahien — Wed, 27 Aug 2008 19:39:25 GMT

I am just going to post that, and watch what happens. I will note that this is code that I just wrote, from scratch.

public class TaxCalculator
{
    private string conStr;
    private DataSet rates;

    public TaxCalculator(string conStr)
    {
        this.conStr = conStr;
        using (SqlConnection con = new SqlConnection(conStr))
        {
            con.Open();
            using (SqlCommand cmd = new SqlCommand("SELECT * FROM tblTxRtes", con))
            {
                rates = new DataSet();
                new SqlDataAdapter(cmd).Fill(rates);
                Log.Write("Read " + rates.Tables[0].Rows.Count + " rates from database");
                if (rates.Tables[0].Rows.Count == 0)
                {
                    MailMessage msg = new MailMessage("important@legacy.org", "joe@legacy.com");
                    msg.Subject = "NO RATES IN DATABASE!!!!!";
                    msg.Priority = MailPriority.High;
                    new SmtpClient("mail.legacy.com", 9089).Send(msg);
                    Log.Write("No rates for taxes found in " + conStr);
                    throw new ApplicationException("No rates, Joe forgot to load the rates AGAIN!");
                }
            }
        }
    }

    public bool Process(XmlDocument transaction)
    {
        try
        {
            Hashtable tx2tot = new Hashtable();
            foreach (XmlNode o in transaction.FirstChild.ChildNodes)
            {
            restart:
                if (o.Attributes["type"].Value == "2")
                {
                    Log.Write("Type two transaction processing");
                    decimal total = decimal.Parse(o.Attributes["tot"].Value);
                    XmlAttribute attribute = transaction.CreateAttribute("tax");
                    decimal r = -1;
                    foreach (DataRow dataRow in rates.Tables[0].Rows)
                    {
                        if ((string)dataRow[2] == o.SelectSingleNode("//cust-details/state").Value)
                        {
                            r = decimal.Parse(dataRow[2].ToString());
                        }
                    }
                    Log.Write("Rate calculated and is: " + r);
                    o.Attributes.Append(attribute);
                    if (r == -1)
                    {
                        MailMessage msg = new MailMessage("important@legacy.org", "joe@legacy.com");
                        msg.Subject = "NO RATES FOR " + o.SelectSingleNode("//cust-details/state").Value + " TRANSACTION !!!!ABORTED!!!!";
                        msg.Priority = MailPriority.High;
                        new SmtpClient("mail.legacy.com", 9089).Send(msg);
                        Log.Write("No rate for transaction in tranasction state");
                        throw new ApplicationException("No rates, Joe forgot to load the rates AGAIN!");
                    }
                    tx2tot.Add(o.Attributes["id"], total * r);
                    attribute.Value = (total * r).ToString();
                }
                else if (o.Attributes["type"].Value == "1")
                {
                    //2006-05-02 just need to do the calc
                    decimal total = 0;
                    foreach (XmlNode i in o.ChildNodes)
                    {
                        total += ProductPriceByNode(i);
                    }
                    try
                    {
                        // 2007-02-19 not so simple, TX has different rule
                        if (o.SelectSingleNode("//cust-details/state").Value == "TX")
                        {
                            total *= (decimal)1.02;
                        }
                    }
                    catch (NullReferenceException)
                    {
                        XmlElement element = transaction.CreateElement("state");
                        element.Value = "NJ";
                        o.SelectSingleNode("//cust-details").AppendChild(element);
                    }
                    XmlAttribute attribute = transaction.CreateAttribute("tax");
                    decimal r = -1;
                    foreach (DataRow dataRow in rates.Tables[0].Rows)
                    {
                        if ((string)dataRow[2] == o.SelectSingleNode("//cust-details/state").Value)
                        {
                            r = decimal.Parse(dataRow[2].ToString());
                        }
                    }
                    if (r == -1)
                    {
                        MailMessage msg = new MailMessage("important@legacy.org", "joe@legacy.com");
                        msg.Subject = "NO RATES FOR " + o.SelectSingleNode("//cust-details/state").Value + " TRANSACTION !!!!ABORTED!!!!";
                        msg.Priority = MailPriority.High;
                        new SmtpClient("mail.legacy.com", 9089).Send(msg);
                        throw new ApplicationException("No rates, Joe forgot to load the rates AGAIN!");
                    }
                    attribute.Value = (total * r).ToString();
                    tx2tot.Add(o.Attributes["id"], total * r);
                    o.Attributes.Append(attribute);
                }
                else if (o.Attributes["type"].Value == "@")
                {
                    o.Attributes["type"].Value = "2";
                    goto restart;
                    // 2007-04-30 some bastard from northwind made a mistake and they have 3 months release cycle, so we have to
                    // fix this because they won't until sep-07
                }
                else
                {
                    throw new Exception("UNKNOWN TX TYPE");
                }
            }
            SqlConnection con2 = new SqlConnection(conStr);
            SqlCommand cmd2 = new SqlCommand();
            cmd2.Connection = con2;
            con2.Open();
            foreach (DictionaryEntry d in tx2tot)
            {
                cmd2.CommandText = "usp_TrackTxNew";
                cmd2.Parameters.Add("cid", transaction.SelectSingleNode("//cust-details/@id").Value);
                cmd2.Parameters.Add("tx", d.Key);
                cmd2.Parameters.Add("tot", d.Value);
                cmd2.ExecuteNonQuery();
            }
            con2.Close();
        }
        catch (Exception e)
        {
            if (e.Message == "UNKNOWN TX TYPE")
            {
                return false;
            }
            throw e;
        }
        return true;
    }

    private decimal ProductPriceByNode(XmlNode item)
    {
        using (SqlConnection con = new SqlConnection(conStr))
        {
            con.Open();
            using (SqlCommand cmd = new SqlCommand("SELECT * FROM tblProducts WHERE pid=" + item.Attributes["id"], con))
            {
                DataSet set = new DataSet();
                new SqlDataAdapter(cmd).Fill(set);
                return (decimal)set.Tables[0].Rows[0][4];

            }
        }
    }
}

Legacy Driven Development

Ayende Rahien — Wed, 27 Aug 2008 19:37:30 GMT

Here is an interesting problem that I run into. I needed to produce an XML document for an external system to consume. This is a fairly complex document format, and there are a lot of scenarios to support. I began to test drive the creation of the XML document, but it turn out that I kept having to make changes as I run into more scenarios that invalidated previous assumptions that I made.

Now, we are talking about a very short iteration cycle, I might write a test to validate an assumption (attempting to put two items in the same container should throws) and an hour later realize that it is a legal, if strange, behavior. The tests became a pain point, I had to keep updating things because the invariant that they were based upon were wrong.

At that point, I decided that TDD was exactly the wrong approach for this scenario. Therefor, I decided that I am going to fall back to the old "trial and error" method. In this case, producing the XML and comparing using a diff tool.

The friction in the process went down significantly, because I didn't have to go and fix the tests all the time. I did break things that used to work, but I caught them mostly with manual diff checks.

So far, not a really interesting story. What is interesting is what happens when I decided that I have done enough work to consider most scenarios to be completed. I took all the scenarios and started generating tests for those. So for each scenario I now have a test that tests the current behavior of the system. This is blind testing. That is, I assume that the system is working correctly, and I want to ensure that it keeps working in this way. I am not sure what each test is doing, but the current behavior is assumed to be correct until proven otherwise..

Now I am back to having my usual safety net, and it is a lot of fun to go from zero tests to nearly five hundred tests in a few minutes.

This doesn't prove that the behavior of the system is correct, but it does ensure no regression and make sure that we have a stable platform to work from. We might find a bug, but then we can fix it in safety.

I don't recommend this approach for general use, but for this case, it has proven to be very useful.

re: MVC Storefront Part 19: Processing Orders With Windows Workflow

Ayende Rahien — Sun, 17 Aug 2008 02:59:59 GMT

Rob Conery has another MVC Storefront post, this time focusing on using Windows Workflow.

Those are my random impressions:

You probably do want to test your work flow. In the same way you want to have an integration test for the system.
The sequence work flow seems to be a very heavy weight approach to just orchestrating actions in the application.
I wonder what the perf implications of creating a workflow here would be. My gut feeling is that this is not good, but I don't really have data for that.
There is probably an issue here with the WF being run in async, I am not sure where it is getting its threads, but if it is from the thread pool, then it is consuming request handling threads, which can kill a site.

As an aside, here is the checkout workflow:

And here is how I would write this:

ValidateOrder()
AuthorizePayment()
order.Status = OrderStatus.Verified
SaveOrder()

Much easier, I think :-) And even more flexible.

3 Levels Of Failures

Ayende Rahien — Fri, 01 Aug 2008 13:24:51 GMT

You have the no-failure, everything completed successfully and everyone is happy.
You have complete-failure, someone just pull the power cord off the machine.
And in the middle, you have partial-failure, something bad happened, (network is down, someone pulled the USB key out) but the system can (and should) keep running.

Most people give up on the complete failure scenario, leaving that up to transactional storage to handle this. That is usually the smartest solution.

The problem occur with partial failure, where you need to recover in meaningful ways. It is actually very easy to write a system where a complete failure is better than a partial failure.

Useless Java

Ayende Rahien — Wed, 30 Jul 2008 13:36:18 GMT

This article (The 'Anti-Java' Professor and the Jobless Programmers) has really annoyed me. Not because of its content, but because the professor being interview is so narrowly focused that he is completely out of it.

Let us take this quote:

“Furthermore, Java is mainly used in Web applications that are mostly fairly trivial,” Dewar says, with his characteristic candor. “If all we do is train students to be able to do simple Web programming in Java, they won't get jobs, since those are the jobs that can be easily outsourced. What we need are software engineers who understand how to build complex systems.”

You know what, I might agree with this, trivial web programming is something that doesn't require highly skilled worker. But the context in which he says this makes all the difference in the world. Here is the next statement:

“By the way Java has almost no presence in such systems. At least as of a few months ago, there was not a single line of safety-critical Java flying in commercial or military aircraft. I recently talked to a customer who had a medium-sized application in Java, which was to be adapted to be part of a safety-critical avionics system. They immediately decided that this meant it would have to be recoded in a suitable language, probably either C or Ada.”

So, if it is not aircraft code, it is "simple Web programming"? Excuse me?!

Leaving aside the part about Java's license not being suitable for life critical systems, it annoys me that this is the only criteria that he think is worth mentioning. His bank is running Java, as well as the stock exchange, it is likely that the payroll system in his university as well. Relegating Java to "simple Web programming" is a huge mistake. Java sweet spot is building complex systems, where a lot of the design decisions that make it hard to do the simple stuff pay off tremendously.

I can think of quite a few mission critical (people life on the balance) that are written in C# (for the purpose of this post, equivalent to Java). That is not to say that aircraft systems shouldn't be written in Ada or C, I never written one, so I won't comment, but to say that this is the only field of any complexity is showing a remarkable amount of ignorance and bigotry.

There was another part that caught my eye, this professor interviewing's technique:

Dewar says that if he were interviewing applicants for a development job, he would quickly eliminate the under-trained by asking the following questions:

1.) You begin to suspect that a problem you are having is due to the compiler generating incorrect code. How would you track this down? How would you prepare a bug report for the compiler vendor? How would you work around the problem?

2.) You begin to suspect that a problem you are having is due to a hardware problem, where the processor is not conforming to its specification. How would you track this down? How would you prepare a bug report for the chip manufacturer, and how would you work around the problem?

“I am afraid I would be met by blank stares from most recent CS graduates, many of whom have never seen assembly or machine language!” he says

I can tell quite a lot about what he is doing, just by the questions he ask, but let me try answering them.

1) Well, I would start by creating an isolated test case, seeing if I can still see the odd behavior. I would remove all code that is even half way smart, trying to reduce the amount of work that the compiler has to do. I would get the language spec and pore over it, trying to make sure that I am not relying on unspecified behavior or misunderstanding the spec. Hopefully, I have a decompiler as well, since that would be an independent corroboration that there is or isn't an issue. Then I would take the output and verify disassemble it, comparing it to what is supposed to happen. I will take that info and submit to the compiler vendor. Working around this issue... by now I have some understanding on what triggers it, so I would change the code accordingly, simplifying it, removing any smart tricks, etc. I'll also leave a big comment about why this is done.

2) Try it on a different machine with the same processor, try it on another machine with a different processor. That rules problems with a specific machine and problems with my code, respectively. Reduce the problem to the smallest thing that can repro it, go over the spec, verify understanding. Eventually I should have a small program that demonstrate the issue. This can be incredibly hard to do if you run into something like memory barriers do not work in some scenario on SMP machines, for example. Work around it... depending on context...

Not all objects are created equals

Ayende Rahien — Mon, 28 Jul 2008 22:53:52 GMT

I found something extremely surprising while profiling a project. Take a look at this piece of code:

Stopwatch stop = Stopwatch.StartNew();
for (int i = 0; i < 1000000; i++)
{
	new WeakReference(null);
}
stop.Stop();
Console.WriteLine("WeakRef: " + stop.ElapsedMilliseconds);

stop = Stopwatch.StartNew();
for (int i = 0; i < 1000000; i++)
{
	new string('a', 5);
}
stop.Stop();
Console.WriteLine("'aaaaa': " + stop.ElapsedMilliseconds);

On my machine, this has the following output:

WeakRef: 980
'aaaaa': 35

Creating a WeakReference is much more costly than creating a normal object. Not surprising, when you think of it, WeakReference has deep ties to the CLR, but I couldn't really believe it when I saw it the first time.

Testing interleaved async file writing

Ayende Rahien — Sat, 26 Jul 2008 19:47:09 GMT

Here is another interesting aspect that I run into when thinking about writing to files. What will happen if I start two async write operations? Will they get interleaved? Will they get interleaved? Will I get errors when using this?

Here is using a single stream:

var path = "test.txt";

if (File.Exists(path))
	File.Delete(path);

var handles = new List<WaitHandle>();
using (
	var stream = new FileStream(path, FileMode.CreateNew, FileAccess.Write, FileShare.None, 0x1000,
								 FileOptions.Asynchronous))
{
	for (int i = 0; i < 64; i++)
	{
		var handle = new ManualResetEvent(false);
		var bytes = Encoding.UTF8.GetBytes( i + Environment.NewLine);
		stream.BeginWrite(bytes, 0, bytes.Length, delegate(IAsyncResult ar)
		{
			stream.EndWrite(ar);
			handle.Set();
		}, stream);
		handles.Add(handle);
	}
	WaitHandle.WaitAll(handles.ToArray());
	stream.Flush();

}

This worked perfectly, the writes are written in a sequential manner, with no interleaving.

The situation changes somewhat when we work with several streams, instead. Here is the code:

var path = "test.txt";

if (File.Exists(path))
	File.Delete(path);

var handles = new List<WaitHandle>();

for (int i = 0; i < 64; i++)
{
	var stream = new FileStream(path, FileMode.OpenOrCreate, FileAccess.Write, FileShare.ReadWrite, 0x1000,
	                            FileOptions.Asynchronous);
	var handle = new ManualResetEvent(false);
	var bytes = Encoding.UTF8.GetBytes(i + Environment.NewLine);
	stream.BeginWrite(bytes, 0, bytes.Length, delegate(IAsyncResult ar)
	{
		stream.EndWrite(ar);
		stream.Flush();
		stream.Dispose();
		handle.Set();
	}, stream);
	handles.Add(handle);
}
WaitHandle.WaitAll(handles.ToArray());

The output of this code on my machine is: 63

Although I am pretty sure that it is possible to get other things as well. But considering that the way IO operations work in generate ( Write( position, buffer) ), that makes a lot of sense.

Interestingly, in regards to the single stream version, the documentation states:

Multiple simultaneous asynchronous requests render the request completion order uncertain.

It doesn't say anything about the actual write order, however, the completion order is not an issue, as far as I am concerned, but the order of the data on disk is very important.

Another interesting question was how a single file stream work with multiple threads calling BeginWrite in parallel. Here is the code:

var path = "test.txt";

if (File.Exists(path))
	File.Delete(path);

var handles = new List<WaitHandle>();
using (
	var stream = new FileStream(path, FileMode.CreateNew, FileAccess.Write, FileShare.None, 0x1000,
								 FileOptions.Asynchronous))
{
	for (int i = 0; i < 64; i++)
	{
		var handle = new ManualResetEvent(false);
		var bytes = Encoding.UTF8.GetBytes(i + Environment.NewLine);
		ThreadPool.QueueUserWorkItem(delegate
		{
			Thread.Sleep(10);
			stream.BeginWrite(bytes, 0, bytes.Length, delegate(IAsyncResult ar)
			{
				stream.EndWrite(ar);
				handle.Set();
			}, stream);
		});
		handles.Add(handle);
	}
	WaitHandle.WaitAll(handles.ToArray());
	stream.Flush();

}

The result of this code looks like this:

Obviously, we have a race condition with the known position of the stream. Adding a lock for the stream.BeginWrite results in this:

I am actually fine with this, because as far as I can tell, this might very well be the actual execution order.

Another observation that is very important is the fact that the Position property of the stream is update immediately when you can BeginWrite().

That was a somewhat random walk over the BeginWrite implementation, I admit, but I hope it will make sense soon.

In search of an embedded DB

Ayende Rahien — Thu, 03 Jul 2008 08:04:48 GMT

I need to use an embedded DB in a project. Naturally, I turned to my old friend, SQLite. It has served me well in the past, and I am well versed in all its tricks. Except... SQLite really fall apart when you are talking about even moderately multi threaded applications. By that I mean, it works just as advertised, for sure. It just lock the entire DB when used, which tend to kill other threads who want to access the poor DB.

I then turned to FireBird, but a short spike told me that it is... inadequate. I managed to get the database into inconsistent state, but I am not sure how or why. There seem to be very little information about it as well, so at the moment it is out of the game.

Sql CE is worse than SQLite when it comes to handling multiple threads, so that wasn't even in the running.

Vista DB is commercial, and I plan to make this an OSS project, so that is out as well.

I want to use Berkeley DB, but is seems like the .Net story is hard. There are CLR Binding, but I run into several blocking issues with them. There is no 1:1 mapping to the C API or the Java API, the error messages are... terse. And there aren't any tests at all! Even worse, the code is interfacing with native code, so it is C# after a rail road has hit it at 200 kph and having spent three days crying in a desert.

I put some time into trying to get this to work, and once the initial hump is over (understanding the association between environments, databases and transactions), it makes a lot of sense. Of course, there are a lot of features that I wouldn't even try to figure out.

First order of business, create a .Netified wrapper that will abstract the C like API away from me.

Okay, I am being ungracious here, the API make a lot of sense and the code is good for the purpose it was built, it was just that I had a moment of existential uncertainty when I looked at the code and it looked like C++#.