Bugs

How do you track that?

Ayende Rahien — Mon, 05 May 2008 08:09:12 GMT

I have an interesting problem with SvnBridge.

After around 5000 full revision request (a set of requests that can occur), the application get hung making a web service call to TFS. This comes after making quite a few calls to TFS, and is generally fairly easily reproducible. The actual call being made is not an expensive one (nor is it the same call). TFS is responsive during that time, so it is not its fault.

It looks very much like I am hitting the 2 concurrent HTTP requests, except that all requests are serialized, and there is no multi threaded work involved.

I have been unable to reproduce this under a profiler or debugger...

Thoughts?

Source control is not a feature you can postpone to vNext

Ayende Rahien — Thu, 17 Apr 2008 08:57:10 GMT

I was taking part in a session in the MVP Summit today, and I came out of it absolutely shocked and bitterly disappointed with the product that was under discussion. I am not sure if I can talk about that or not, so we will skip the name and the purpose. I have several issues with the product itself and its vision, but that is beside the point that I am trying to make now.

What really bothered me is utter ignorance of a critical requirement from Microsoft, who is supposed to know what they are doing with software development. That requirement is source control.

Source control is not a feature
Source control is a mandatory requirement

The main issue is that the product uses XML files as its serialization format. Those files are not meant for human consumption, but should be used only through a tool. The major problem here is that no one took source control into consideration when designing those XML files, so they are unmergable.

Let me give you a simple scenario:

Developer A makes a change using the tool, let us say that he is modifying an attribute on an object.
Developer B makes a change using the tool, let us say tat he is modifying a different attribute on a different object.

The result?

Whoever tries to commit last will get an error, the file was already updated by another guy. Usually in such situations you simply merge the two versions together, and don't worry about this.

The problem is that this XML file is implemented in such a way that each time you save it, a whole bunch of stuff gets moved around, all sort of unrelated things change, etc. In short, even a very minor change cause a significant change in the underlying XML.

You can see this in products that are shipping today, like SSIS, WF, DSL Toolkit, etc.

The problem is that when you try to merge, you have too many unrelated changes, which completely defeat the purpose of merging.

This, in turn, means that you lose the ability to work in a team environment. This product is supposed to be aimed at big companies. But it can't suppose a team of more than one! To make things worse, when I brought up this issue, the answer was something along the line: "Yes, we know about this issue, but you can avoid this using exclusive checkouts."

At that point, I am not really sure what to say. Merges happen not just when two developers modify the same file, merges also happen when you have branches. As a simple scenario, I have a development branch and a production branch. Fixing a bug in the production branch requires touching this XML file. But if I made any change to it on the development branch, you can't merge that. What happen if I use a feature branch? Or version branches?

Not considering the implications of something as basic as source control is amateurish in the extreme. Repeating the same mistake, over and over again, across multiple products, despite customer feedback on how awful this is and how much it hurt the developers who are going to use it shows contempt to the end developers, and a sign of even more serious issue: the team isn't dogfooding the product. Not in any real capacity. Otherwise, they would have already noticed the issue, much sooner in the lifetime of the product, with enough time to actually fix that.

As it was, I was told that there is nothing to do for the v1 release, that puts the fix (at best) in two years or so. For something that is a complete deal breaker for any serious development.

I have run into issues where merge issues with SSIS caused us to have to drop days of work and having to recreating everything from scratch, costing us something in the order of two weeks. I know of people that had the same issue with WF, and from my experiments, the DSL toolkit has had the exact same issue. The SSIS issues were initially reported on 2005, but are not going to be fixed for the 2008 (or so I heard from public sources) , which puts the nearest fix for something as basic as getting source control right in a 2-3 years time.

The same for the product that I am talking about here. I am not really getting that, does Microsoft things that source control isn't an important issue? They keep repeating this critically serious mistake!

For me, this is unprofessional behavior of the first degree.

Deal breaker, buh-bye.

The illustrated guide to confused concurrency

Ayende Rahien — Wed, 09 Apr 2008 23:15:42 GMT

This has me very confused.

The answer, however, turned out to be that I am an idiot:

Brute force concurrency testing

Ayende Rahien — Wed, 09 Apr 2008 14:07:37 GMT

Testing concurrency is hard, just ask anyone who had tried. And tracking down concurrent bugs is even harder.

I just run into concurrency issue with SvnBridge, and I decided that I have better have a smoke test around to prove that there are no obvious concurrency issues with the code.

Since I already have over a hundred integration tests, I thought that this would make a great concurrency test. We know that all the tests are passing. Now let us see if they are all passing together. This is quick & dirty, but it expose at least two bugs so far, one of them the original one that we have seen:

public class ConcurrentActionsTest
{
	[Fact]
	public void RunAllTestsConcurrentyly()
	{
		List<TestData> tests = new List<TestData>();
		Type[] types = Assembly.GetExecutingAssembly().GetTypes();
		foreach (Type type in types)
		{
			if (type.IsAbstract)
				continue;
			if (type == typeof(ConcurrentActionsTest))
				continue;
			foreach (MethodInfo info in type.GetMethods())
			{
				object[] attributes = info.GetCustomAttributes(typeof(SvnBridgeFactAttribute), true);
				if (attributes.Length == 0)
					continue;
				tests.Add(new TestData((SvnBridgeFactAttribute)attributes[0], info));
			}
		}

		List<IAsyncResult> results = new List<IAsyncResult>();
		List<Exception> errors = new List<Exception>();
		ExecuteTestDelegate exec = ExecuteTest;
		foreach (TestData test in tests)
		{
			foreach (ITestCommand command in test.Fact.CreateTestCommands(test.Method))
			{
				IAsyncResult invoke = exec.BeginInvoke(test, command, errors, null, null);
				results.Add(invoke);
			}
		}
		foreach (IAsyncResult result in results)
		{
			result.AsyncWaitHandle.WaitOne();
			exec.EndInvoke(result);
		}
		if (errors.Count > 0)
		{
			StringBuilder sb = new StringBuilder();
			foreach (Exception error in errors)
			{
				sb.AppendLine(error.ToString());
			}
			throw new Exception(sb.ToString());
		}
	}

	private delegate void ExecuteTestDelegate(TestData test, ITestCommand command, List<Exception> errors);

	private void ExecuteTest(TestData test, ITestCommand command, List<Exception> errors)
	{
		try
		{
			object instance = Activator.CreateInstance(test.Method.DeclaringType);
			command.Execute(instance);
		}
		catch (TargetInvocationException e)
		{
			lock (errors)
				errors.Add(e.InnerException);
		}
		catch (Exception e)
		{
			lock (errors)
				errors.Add(e);
		}
	}

	private class TestData
	{
		public readonly MethodInfo Method;
		public readonly SvnBridgeFactAttribute Fact;

		public TestData(SvnBridgeFactAttribute fact, MethodInfo method)
		{
			Fact = fact;
			Method = method;
		}
	}
}

As I was writing this post, I figure out one issue, the other would require concurrent debugging...

SQL CE Issues, Part 2

Ayende Rahien — Tue, 08 Apr 2008 15:03:22 GMT

Hopefully I'll get the same quick "you are an idiot, this is how it is done" that I got the first time I posted about it. Here is my current issue. Attempting to open SQL CE from multiple threads has locked it out. This has been the state of the system for ~6 hours.

I don't mind the locks, I do mind the fact that there seems to be no way to specify a timeout for that, so the app just sit there, waiting, waiting, waiting.

Internals are Evil

Ayende Rahien — Thu, 03 Apr 2008 01:45:15 GMT

A while ago I talked about Internal Stripping, not surprisingly, I run into another "WTF is this internal?" class just now. And I think it would be a good sample to see what can cause me to pull the pliers and start messing with things.

The scenario was simple, I was talking with Rob Conery about a webcast he is doing, and we started talking about TDD and repositories in a Linq enabled world.

What we ended up with was the requirement to turn an List<T> to IQueryable<T>, and there isn't easy way to do that. The framework does have an implementation of this, but, naturally, it is marked internal. Not content to have to deal with writing a non trivial Linq to List<T> provider, I wrote the following:

private static IQueryable<T> CreateQueryable<T>(IEnumerable<T> inner)
{
   Type queryable = typeof (IQueryable).Assembly.GetType("System.Linq.EnumerableQuery`1").MakeGenericType(typeof (T));
   ConstructorInfo constructor = queryable.GetConstructor(BindingFlags.Instance | BindingFlags.NonPublic, 
                    null, new Type[] {typeof (IEnumerable<T>)},new ParameterModifier[0]);
   object instance = FormatterServices.GetSafeUninitializedObject(queryable);
   constructor.Invoke(instance, new object[] { inner });
   return (IQueryable<T>)instance;
}

Not it works.

Why I hate internals, yet again

Ayende Rahien — Sun, 09 Mar 2008 01:26:16 GMT

So here I am searching for an implementation of a WeakHashtable, and imagine my surprise when I find out that the BCL already has one such implementation. Imagine my lack of surprise to find that System.ComponentModel.WeakHashtable is internal, thus rendering it near useless to me.So annoying, frustrating and irritating.
Update:
Oh, and as if it is not enough, it appears that there are two implementation of WeakHashtable in the BCL. In two different assemblies (System and System.Windows.Forms).
From browsing the code, it looks like they just copy/paste the code, instead, you know, recognizing that this is an actual need and exposing this.

TFS, SOAP and XMLish

Ayende Rahien — Thu, 06 Mar 2008 03:53:17 GMT

Today I discovered some really interesting issue with TFS. I was sending it the following message, and no, you are not seeing double, it was doubled. That was the XML payload that I was sending to a web service.

Now, the crazy thing about this, it worked. And that really scares me.

You could argue that SVN's behavior is correct, but there is not way that this can be considered a valid XML document.

<?xml version="1.0" encoding="utf-8"?>

<soap:Envelope

    xmlns:soap="http://www.w3.org/2003/05/soap-envelope"

    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"

    xmlns:xsd="http://www.w3.org/2001/XMLSchema">

    <soap:Header>

        <RequestHeader

            xmlns="http://schemas.microsoft.com/TeamFoundation/2005/06/WorkItemTracking/ClientServices/03">

            <Id>uuid</Id>

        </RequestHeader>

    </soap:Header>

    <soap:Body>

        <GetWorkItem

            xmlns="http://schemas.microsoft.com/TeamFoundation/2005/06/WorkItemTracking/ClientServices/03">

            <workItemId>2192</workItemId>

            <revisionId>0</revisionId>

            <minimumRevisionId>0</minimumRevisionId>

            <asOfDate xsi:nil="true" />

            <useMaster>true</useMaster>

        </GetWorkItem>

    </soap:Body>

</soap:Envelope>

<?xml version="1.0" encoding="utf-8"?>

<soap:Envelope

    xmlns:soap="http://www.w3.org/2003/05/soap-envelope"

    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"

    xmlns:xsd="http://www.w3.org/2001/XMLSchema">

    <soap:Header>

        <RequestHeader

            xmlns="http://schemas.microsoft.com/TeamFoundation/2005/06/WorkItemTracking/ClientServices/03">

            <Id>uuid</Id>

        </RequestHeader>

    </soap:Header>

    <soap:Body>

        <GetWorkItem

            xmlns="http://schemas.microsoft.com/TeamFoundation/2005/06/WorkItemTracking/ClientServices/03">

            <workItemId>2192</workItemId>

            <revisionId>0</revisionId>

            <minimumRevisionId>0</minimumRevisionId>

            <asOfDate xsi:nil="true" />

            <useMaster>true</useMaster>

        </GetWorkItem>

    </soap:Body>

</soap:Envelope>

TCP Errors: Tracking it down

Ayende Rahien — Thu, 06 Mar 2008 02:41:38 GMT

I get to the office this morning, determined to figure out what the hell was going on with my TCP stack. And I can reproduce the error in about 1/3 of the cases.
I know that this is my problem, so I decide to move to a known good version and try to figure out where the error happens. I am doing that, and eventually I get to the head of the repository, without being able to reproduce the error.
After calling for a check of the tooth fairy and requesting immediate assistance from an experienced exorcist, I was unable to reproduce the error, which I found to be really annoying.
I spent some time trying to reproduce that, but couldn't, so I moved to other things.
After an hour, I needed to figure what was going on at the network layer, so I opened Fiddler. A minute afterward I got the same error.
Now I had someone to blame, hurray!
Indeed, it looks like Fiddler isn't quite up to speed for handling large amount of connections in short amount of time. I am not quite sure what is going on there. The scary thing is that I tested that on 4 different machines, and in all cases, I opened Fiddler first to ensure that I knew what was going on.

How I broke the TCP Stack

Ayende Rahien — Wed, 05 Mar 2008 07:45:09 GMT

Today I spend a large part of the day tracking down various strange errors. Those errors would be of the following variety:

The connection could not be made because the target machine actively refused it.
A connect request was made on an already connected socket.
The underlying connection was closed: A connection that was expected to be kept alive was closed by a server.

Now, I am not doing anything strange with TCP/IP, I am just making plain old web service calls (and no, not using my hand rolled SOAP stack).

I can't really figure out what is going on, but I know that the problem is no in the TCP/IP level, it is something in my code, that is breaking things. It doesn't happen always, but it happens often enough.

I have ensured to my satisfaction that the server it up, no firewall in the way, respond to calls, etc.

Flat out weird, and I am not looking forward to trying to figure it out tomorrow.

Any ideas?