Debugging

How do you track that?

Ayende Rahien — Mon, 05 May 2008 08:09:12 GMT

I have an interesting problem with SvnBridge.

After around 5000 full revision request (a set of requests that can occur), the application get hung making a web service call to TFS. This comes after making quite a few calls to TFS, and is generally fairly easily reproducible. The actual call being made is not an expensive one (nor is it the same call). TFS is responsive during that time, so it is not its fault.

It looks very much like I am hitting the 2 concurrent HTTP requests, except that all requests are serialized, and there is no multi threaded work involved.

I have been unable to reproduce this under a profiler or debugger...

Thoughts?

Continuous Environment Validation

Ayende Rahien — Wed, 27 Feb 2008 16:07:45 GMT

For some reason, it seems like I am talking about Release It a lot lately, to a lot of people. As I said when I reviewed it, that book literally changed the way that I approach problems. It also made me much more aware of the failure scenarios that I need to deal with.

A while ago I sat down in one of Jeremy Miller's talks and he mentioned that he had added the ability to do Environment Validation to StructureMap, so when the application is starting up, it can verify that all its dependencies are in a valid state. That made so much sense to me that I immediately added this facility to Windsor.

What I am going to talk about today is to take this approach one step further. Instead of running those tests just at application startup, they should be run every day, or every hour.

Yes, the operation team is suppose to have monitoring on the application, but unless they were part of the development process (or are a dedicated ops team), that still leaves you as the principal owner of knowledge in about the environment your application need. Even if you have a capable operation team, and they have very good understanding on your application, it is often best to support them by providing this functionality. It is very likely that you can get more information from your application that the operation team.

And if you don't have an operation team, you really want to be able to do that.

Now that we have taken care of the motivation for this approach, let us see what exactly we are talking about.

Environment validation means that you validate the your entire environment is in a state that allows your application to run in full capacity. I am going to list a few things that I think are essential for many applications, I am sure that I am going to miss some, however, feel free to add more items to the list.

Certificate's valid and expire in more than a month.
Domain registration expires in than one month.
For each server in the application (web, database, cache, application):

Server is alive and responding (within specified time).
Server's HD has more than 10% free space.
Server CPU usage is less than 80%

Associated 3rd party servers are responding within their SLA.
Sample execution of common scenarios finish successfully in a specified time frame.
Number of faults (non critical ones) in the application is below the threshold.
No critical fault (critical defined as taking the entire system down).
Current traffic / work on the system is within expected range (too low, and we may have external network issue, too high, and we need to up our capacity).
Application audit trail is updated. (Can do the same for log, if required).
System backup was performed and completed successfully.
All batch jobs have been run and completed successfully.
Verify the previously generated faults has been dealt with.

Those are the generalities, I am pretty sure that you can think of a lot more that fit your own systems.

The important thing to remember here is that you should treat this piece as a core part of the application infrastructure. In many production environment, you simply cannot get access. This is part of the application, and should be deployed with the application. At any rate, it should be made clear that this is part of the deployment program, not just useless appendix.

My preference would be to have a windows service to monitor my systems and alert when there are failures.

This is another important consideration, how do you send alerts? And when? You should have at least three levels of warnings: Warning, Error and Fatal. You send them according to the severity of the problem.

In all cases, I would log them to the event log at a minimum, probably send mail as well. For Error and Fatal levels, I would use SMS / generate alert to operation monitoring systems. If there are monitoring system in place that the operations team is using, it is best to route things through them. They probably have the ability to wake someone up in 3 AM already. If you don't have that, than an SMS is at least near instantaneous, and you can more or less rely on that to be read.

That is long enough, and I have to do some work today, so I'll just stop here, I think.

Exception handling best practices

Ayende Rahien — Thu, 06 Dec 2007 10:05:55 GMT

No, I am not going to tell you to use throw; instead of throw e; I am going to talk about exception messages, assumptions, and pain.

Exception hierarchies are useful in many ways, mostly because they bring order to the way we handle exceptions. We can catch a specific exception, or a root exception in a hierarchy, and hanlde them specifically. But, one of the usages of exception hierarchies is to add additional data to an exception. In many cases, this is very useful data, such as the SQL error code or the details node of in a soap fault or the list of assemblies that could not be loaded.

Do you know what this three datums has in common?

They are very useful
They do not show in ex.ToString()

Guess what is going to be shown in any log, error message, etc?

You got that right, the ex.ToString() output!

If you have additional information in the exception, it must be available on the exception afterward. Trying to diagnose assembly load failures is driving me mad.

Imagine finding things like this in the log:

"ReflectionTypeLoadException: Unable to load one or more of the requested types. Retrieve the LoaderExceptions property for more information."
"SoapException: Server was unable to process request"

Imagine gettting one of those during dev, you have no way of knowing where this is happening. Often you can't even set a breakpoint in the code there, and inspect the exception, because it is handled inside some library code. ASP.Net is a good example of how this can happen, and where this is a highly annoying issue to work with.

To summrise, if you create exceptions, make sure to remember a simple rule, everything should go in the ex.ToString().

A definition of a nightmare platform

Ayende Rahien — Sun, 11 Nov 2007 14:44:22 GMT

Alex has more or less hit on about the worst description of a platform I can think of:

XYZ is a technology of highs and lows... the highs are when you've finally got something to work that should've worked in the first place, the lows are well... all the times in between.

If you are working on such a platform, make yourself happy, just go away.

Amazon's Dynamo

Ayende Rahien — Sat, 06 Oct 2007 15:04:28 GMT

Okay, Amazon has just published an interesting paper about how they manage state for some of their services. The underlying idea is a hash table, distributed, reliable, versioned and simple. They have some interesting constraints that influenced the design of the system, and it is an interesting, if dry, read.

Dare has some comments about it.

I'll limit myself to saying that the data versioning approach is extremely interesting. The idea is that you issue a get(key) and the result is a set of relevant objects that may need reconciliation. They end with a conclusion that while this may seem like putting a lot of the responsibility in the app developers hands, this is something that they already had to deal with due to the scalability requirements that they face.

I wouldn't want to do this for a small site, but I can see the advantages for scaling wide.

Amusingly enough, the classic shopping cart sample appears to be a core service for this system, and a complex one.

The CLR Sources

Ayende Rahien — Wed, 03 Oct 2007 16:04:43 GMT

I have no idea why this isn't in much wider circulation, but this is huge.

ScottGu has announced that Microsoft is Releasing the Source Code for the .NET Framework Libraries.

I am disappointed to see that even in the tiny source code samples that he has in the post I have violent disagreements (they speak about sealing stuff, which I have serious objection to).

This hopefully means a lot less ReflectorDebugging, although I am not sure about all the implication that this has.

Something is terribly wrong here...

Ayende Rahien — Mon, 06 Aug 2007 06:14:57 GMT

And the problem is with me. I kept going over this code, not seeing what was wrong with it, up until I actually read the code. Argh!

Accidental Debugging

Ayende Rahien — Tue, 31 Jul 2007 22:13:22 GMT

I stopped today to take a look at someone's bug, to see if I could help. I stepped away about 6 hours later, after going over a code base I have no familiarity with. Encoding issues when a piece of text is routed through Flash, JSON, C#, UTF 9 - 23 and other nice stuff.

Best quote: "Javascript written like C++"

Note to self: String concentration is evil.

I need to get better at evasion techniques :-)

Method Equality

Ayende Rahien — Fri, 27 Jul 2007 10:01:44 GMT

The CLR team deserve a truly great appreciation for making generics works at all. When you get down to it, it is amazingly complex. Most of the Rhino Mocks bugs stems from having to work at that level. Here is one example, comparing method equality. Let us take this simple example:

[TestFixture]
public class WeirdStuff
{
	public class Test<T>
	{
		public void Compare()
		{
			Assert.AreEqual(GetType().GetMethod("Compare"),
				MethodInfo.GetCurrentMethod()
				);
		}
	}

	[Test]
	public void ThisIsWeird()
	{
		new Test<int>().Compare();
	}
}

This is one of those things that can really bites you. And it fails only if the type is a generic type, even though the comparison is made of the closed generic version of the type. Finding the root cause was fairly hard, and naturally the whole thing is internal, but eventually I managed to come up with a way to compare them safely:

private static bool AreMethodEquals(MethodInfo left, MethodInfo right)
{
	if (left.Equals(right))
		return true;
	// GetHashCode calls to RuntimeMethodHandle.StripMethodInstantiation()
	// which is needed to fix issues with method equality from generic types.
	if (left.GetHashCode() != right.GetHashCode())
		return false;
	if (left.DeclaringType != right.DeclaringType)
		return false;
	ParameterInfo[] leftParams = left.GetParameters();
	ParameterInfo[] rightParams = right.GetParameters();
	if (leftParams.Length != rightParams.Length)
		return false;
	for (int i = 0; i < leftParams.Length; i++)
	{
		if (leftParams[i].ParameterType != rightParams[i].ParameterType)
			return false;
	}
	if (left.ReturnType != right.ReturnType)
		return false;
	return true;
}

The secret here is with the call to GetHashCode, which remove the method instantiation code, which is fairly strange concept, because I wasn't aware that you can instantiate methods :-)

Debugging NHibernate

Ayende Rahien — Mon, 04 Jun 2007 13:48:18 GMT

Today we had a problem with an NHibernate query that was failing which had quite stumped. Pulling the usual tricks didn't work, debugging NHibernate was problematic, since the failing query was damn complex, and I had no clear idea why this was failing. After a while, I decided that trying the top down approach will not work, and that I need more structure in finding out the issue.

Did I mentioned that the query was complex? The object model is big as well, and the query managed to touch just about all of it. Getting it slimmed down to a reproducable version was hard, because I wasn't sure what caused the issue, but eventually I managed to get it to fail the way I wanted it. (In the process I walked through parts of NHibernate that I haven't met before, interesting.

The end result are these tests (there is a reason that I know that C# has a 512 characters limits on identifiers):

CanMakeCriteriaQueryAcrossBothAssociationsWhenBothAssoicationsHasSameColumnKeyNameAndUsingPagingInSqlServer2005
CanLoadCollectionUsingLeftOuterJoinWhenBothCollectionsHasTheSameColumnKeyNameAndOneIsNull

I still don't have an answer, but now I have much harder questions...

This post is dedicated to Rinat, who says that she doesn't knows NHibernate, but can make queries so complex both NHibernate and SQL Server begs for mercy.