Testing concurrency is hard, just ask anyone who had tried. And tracking down concurrent bugs is even harder.

I just run into concurrency issue with SvnBridge, and I decided that I have better have a smoke test around to prove that there are no obvious concurrency issues with the code.

Since I already have over a hundred integration tests, I thought that this would make a great concurrency test. We know that all the tests are passing. Now let us see if they are all passing together. This is quick & dirty, but it expose at least two bugs so far, one of them the original one that we have seen:

public class ConcurrentActionsTest
{
	[Fact]
	public void RunAllTestsConcurrentyly()
	{
		List<TestData> tests = new List<TestData>();
		Type[] types = Assembly.GetExecutingAssembly().GetTypes();
		foreach (Type type in types)
		{
			if (type.IsAbstract)
				continue;
			if (type == typeof(ConcurrentActionsTest))
				continue;
			foreach (MethodInfo info in type.GetMethods())
			{
				object[] attributes = info.GetCustomAttributes(typeof(SvnBridgeFactAttribute), true);
				if (attributes.Length == 0)
					continue;
				tests.Add(new TestData((SvnBridgeFactAttribute)attributes[0], info));
			}
		}

		List<IAsyncResult> results = new List<IAsyncResult>();
		List<Exception> errors = new List<Exception>();
		ExecuteTestDelegate exec = ExecuteTest;
		foreach (TestData test in tests)
		{
			foreach (ITestCommand command in test.Fact.CreateTestCommands(test.Method))
			{
				IAsyncResult invoke = exec.BeginInvoke(test, command, errors, null, null);
				results.Add(invoke);
			}
		}
		foreach (IAsyncResult result in results)
		{
			result.AsyncWaitHandle.WaitOne();
			exec.EndInvoke(result);
		}
		if (errors.Count > 0)
		{
			StringBuilder sb = new StringBuilder();
			foreach (Exception error in errors)
			{
				sb.AppendLine(error.ToString());
			}
			throw new Exception(sb.ToString());
		}
	}

	private delegate void ExecuteTestDelegate(TestData test, ITestCommand command, List<Exception> errors);

	private void ExecuteTest(TestData test, ITestCommand command, List<Exception> errors)
	{
		try
		{
			object instance = Activator.CreateInstance(test.Method.DeclaringType);
			command.Execute(instance);
		}
		catch (TargetInvocationException e)
		{
			lock (errors)
				errors.Add(e.InnerException);
		}
		catch (Exception e)
		{
			lock (errors)
				errors.Add(e);
		}
	}

	private class TestData
	{
		public readonly MethodInfo Method;
		public readonly SvnBridgeFactAttribute Fact;

		public TestData(SvnBridgeFactAttribute fact, MethodInfo method)
		{
			Fact = fact;
			Method = method;
		}
	}
}

As I was writing this post, I figure out one issue, the other would require concurrent debugging...

Rob Conery has been doing a screen cast series, showing how to build an application using MS MVC. He has also decided to not only take care of application building, but do it in a way that many members of the community, myself included, feel passionately about. Using TDD, DDD concepts, etc.

Rob freely admits that he is exploring a lot of ground as he is producing those screen casts, and I think that he is doing a good job.

There were several instances of overly harsh critique about the screen casts, which do injustice to the work being done. Don't treat this as the Final Guidance, treat this as a journey, and pitch in if you have interesting commentary.

Rob has been very open about it, and I, for one, am very happy about it.

A few days ago I posted about two phase tests, I have been thinking about this lately, and decided that I have a good example for this, which also demonstrate some important design decisions.

The task is listing the first 10 products that we can sell to a customer. The UI is console application, and the database design and data access method are whatever you want.

That is pretty easy, right?

Expected input is:

/dev/product_listing/bin> list_products

Expected output is:

Milk                  $1.0
Bread               $1.3
Sausage            $2.5
Horror Movie    $5.0

Next requirement is that given the following input:

/dev/product_listing/bin> list_products -pg13

The output is:

Milk                  $1.0
Bread               $1.3
Sausage            $2.5

Next requirement is that given the following input:

/dev/product_listing/bin> list_products -vegetarian

Expected output is:

Milk                  $1.0
Bread               $1.3
Horror Movie    $5.0

Additional requirements of this type will follow, and they can be combined. That is, we can also have:

/dev/product_listing/bin> list_products -pg13 -vegetarian

Expected output is:

Milk                  $1.0
Bread               $1.3

How would you solve this?

Hopefully I'll get the same quick "you are an idiot, this is how it is done" that I got the first time I posted about it. Here is my current issue. Attempting to open SQL CE from multiple threads has locked it out. This has been the state of the system for ~6 hours.

I don't mind the locks, I do mind the fact that there seems to be no way to specify a timeout for that, so the app just sit there, waiting, waiting, waiting.

One of the core concepts in NServiceBus is the idea of a Saga. A saga is a stateful class, responsible for handling a set of related messages.

It is important to understand that the saga is not kept in memory between messages, but persisted to storage. This means that we can have large amount of concurrent sagas, without worrying about about memory consumption.

Let us take a real example, from my previous post, and see how a saga can take care of it.

[Serializable, Transactional]
public class CandidateVerificationSaga 
	: ISaga<VerifyCandidateMessage>,
	  ISaga<ReferenceVerificationMessage>,
	  ISaga<CertificationVerificationMessage>
{
	IRepository<IPotentialCandidate> PotentialCandidateRepository { get; set; }
	
	IBus Bus { get; set; }
	
	bool hasRejectedReferencesOrCertifications;
	
	int clientId;
	
	int totalReferences;
	int totalCertifications;
	
	int answeredReferences;
	int answeredCertifications;

	[Transaction]
	public virtual void Handle(VerifyCandidateMessage msg)
	{
		clientId = msg.ClientId;
		
		IPotentialCandidate candidate = PotentialCandidateRepository.Load(msg.CandidateId);
		
		foreach(Reference reference in candidate.References)
		{
			Bus.Send(new ApproveReferenceMessage(candidate, reference));
		}
		
		foreach(Certification cert in candidate.Certifications)
		{
			Bus.Send(new ApproveCertificationMessage(candidate, cert));
		}
		
		totalReferences = candidate.References.Count;
		totalCertifications = candidate.Certifications.Count;
		
		Bus.Send(new TimeoutMessage(msg.AnswerBy, this, null));
	}
	
	[Transaction]
	public virtual void Handle(ReferenceVerificationMessage msg)
	{
		if(msg.Approved == false)
		{
			hasRejectedReferencesOrCertifications = true;
			MarkCandidateAsFailedVerification();
			Bus.Send(new ProblematicReference(candidateId, msg.ReferenceId));
		}
		
		answeredReferences+=1;
		
		CompleteIfDone();
	}
	
	[Transaction]
	public virtual void Handle(CertificationVerificationMessage msg)
	{
		if(msg.Approved == false)
		{
			hasRejectedReferencesOrCertifications = true;
			MarkCandidateAsFailedVerification();
			Bus.Send(new ProblematicCertification(candidateId, msg.CertificationId));
		}
		
		answeredReferences+=1;
		
		CompleteIfDone();
	}
	
	[Transaction]
	public virtual void Timeout(object state)
	{
		Complete();
	}
	
	private void CompleteIfDone()
	{
		if(totalReferences != answeredReferences ||
			totalCertifications != answeredCertifications)
			return;
		
		Complete();
	}
	
	private void Complete()
	{
		bool passedVerification = hasRejectedReferencesOrCertifications == false && 
			(answeredReferences > 0 || answeredCertifications > 0);
		
		if(passedVerification)
			MoveCandidateToAvailableCandidatePool();
		
		Bus.Send(new CandidateVerificationCompleted(candidateId, passedVerification));
		Completed = true;
	}
	
	private void MarkCandidateAsFailedVerification()
	{
		IPotentialCandidate candidate = PotentialCandidateRepository.Load(msg.CandidateId);
		candidate.MarkAsFailedVerification();
	}
	
	private void MoveCandidateToAvailableCandidatePool()
	{
		IPotentialCandidate candidate = PotentialCandidateRepository.Load(msg.CandidateId);
		candidate.MoveToAvailablePool();
	}
}

Notice that we have several entry points into the saga, the Timeout and the Handle(XyzMessage) methods. The saga will handle only a single message at a point in time, so we don't have to worry about concurrency. We also are not going to worry about state, we just store it in instance variables and forget about it.

The programming model is very natural, I believe. We get called when messages that we are interested in arrive, and we correlate between them by the infrastructure, so our code doesn't care about that.

Thoughts?

This question came up in the NServiceBus mailing list, and I think it is very interesting question to try to solve. The standard disclaimers are that I haven't build a real system using this approach, although I am leaning toward it more and more.

I would like to thank Brad for agreeing to post this example here.

I would like some advice as to how to properly identify the services I need to create. I know that Udi has said that creating services for each department is a good starting point, but I keep feeling like something is missing.

So, here's some details.  I work for a company that is the parent of two physician job placement companies.  The parent company would like to create a single system that will support both child companies.  The business processes of both companies have been analyzed and documented, and there is about a 95% overlap in the processes.  Should my team create a monolithic job placement application and wrap it in a service or should I break that application into higher-level business services?

Here's an example of where I get stuck when trying to map the job placement application's functionality to a high-level business service like Sales Service.  The Sales department is broken into two groups: marketers and recruiters.  Marketers call hospitals to find out what positions are available and try to fill these positions.  Recruiters call physicians to find out if the physician is looking for a new job and tries to find available positions.  Before a physician can be placed, his or her credentials must be verified.  The Credentialing department is responsible for doing this.

Ok.  So I now have two service candidates (Sales and Credentialing). I have no issues with the credentialing service but the sales service feels like it should be a job placement service.  What if the business decides to take on a new line of business?  This new line-of-business may have the sales department selling something entirely unrelated to job placement services.  Is a job placement service a better idea? But wouldn't that mean that all job placement functionality would need to be placed in that service?  This feels like I'm loosing the agility that SOA gives the business because I'm no longer modeling the way the departments interact with each other.

I once built a system with very similar concepts, although with drastically different focus. What Brad has already outlined is a good start, but I would break the system even further. We have two main sections of the business that were identified, marketing and recruitment. Let us attack the problem from each side in turn, but keep in mind that we are talking about a single system here.

High level services:

• Available candidates - given a set of requirements, will attempt to find all verified candidates that match the requirements. Additional responsibilities include logging search history (important for noticing what requirements are common and desirable), and recording which requirements has no matching physicians in the system (should be passed for human inspection, to decide if head hunting should begin, or this should be ignored).
• Register open position - enter a new position into the system, with all its requirements. As part of the registration, it will query the available physicians and offer them as matches. If there are no matches, it will register the new position in the recruitment service.
• Register new candidate - enter a new candidate to the system in unverified state. Request credentials verifications from the system.
• Verify credentials - given a candidate, perform a verification of the candidate credentials, references, etc. If the verification succeeds, we move the candidate to the available candidate pool. If the verification fails, we request additional intervention.
• Problematic credentials resolver - there are several reasons for getting problematic credentials, fake credentials, honest mistakes and hostile employers are just a few that comes to mind. In most cases, you need complex logic or human intervention here, and it deserves its own section in the map.
• Candidate hunger* - which candidates are we looking for, this is used by the recruitment guys to know which candidates they should aim for.
• Candidate tracker - if we already assigned a candidate for a job, we already have a happy person, let us keep it this way by ensuring that they remember us, so when they leave, they will contact us again for their next job.
• Potential candidates - given a set of requirements, will attempt to find all candidates that match them. Potential candidates are usually candidates that were gotten from external data sources (the physician listing for the country, for instance), so you want to go through them to see if they can become real candidates.

Those are just the services that I can think of off the top of my head. Notice that each of them has very few responsibilities, and they match pretty closely a specific business scenario. The services communicate between one another, but they do so using one way, async messaging. This is especially important in this type of scenario, where some services can take long time to process a piece of work (verifying credentials requires a human to call and ask for references, for example).

My next post will deal with the actual code for one of those services, for now I think this is enough.

* It was originally a typo, but I like the implications.

I think that I already mentioned that I started to run into scaling limits with my screen casts. More specifically, the latest screen cast, was already downloaded 7,300 times! It is a 160Mb file, it also means that I have to pay for obscene amount of traffic.

Since I didn't want to pay for all that bandwidth, and since in principal I don't believe in having to pay to give something (which cost a lot to make) for free, I was in somewhat of a problem.

The idea of torrents was very attractive, except...

There are no torrents servers for windows. Oh, you can try to use uTorrent using all sorts of hacks using srvany.exe from the resource kits, and hope that it would work (it didn't for me), or you can try using Azureus (still didn't work for me) and pay ~120Mb RAM just for that. The situation seemed hopeless and I toured the halls of the secret laboratory despondent, pondering the situation for many nights.

Then I recalled that I am a developer, therefor I have the power to create things. A quick search on Google led me to BitSharp / MonoTorrent a torrent library in C#. I had to learn how to use it, but it didn't take long, and I wrote my own Torrent Server, which is currently live and used to host my screen casts.

I think that I am doing too much networking stuff lately. Torrent server, subversion server, caching server (see subversion server).

With the advent of the NHibernate Users list, we also needed a place for frequent questions. While the Hibernate's site already has a FAQ in place, I feel that a blog is a better medium for this.

Therefor, welcome the NHibernate FAQ Blog, which Gabriel Schenker and Tom Opgenorth has agreed to maintained. There are several good articles there already. Covering anything from setting up query logging to correct handling of entity equality.

Subscribed.

It is fairly common in job interviews to request some sort of a code sample. Generally a solution for a problem that the prospective employer gives the candidate. An interesting twist on that is to go over the code, give a single additional requirement, then see what the result it.

A while ago I talked about Internal Stripping, not surprisingly, I run into another "WTF is this internal?" class just now. And I think it would be a good sample to see what can cause me to pull the pliers and start messing with things.

The scenario was simple, I was talking with Rob Conery about a webcast he is doing, and we started talking about TDD and repositories in a Linq enabled world.

What we ended up with was the requirement to turn an List<T> to IQueryable<T>, and there isn't easy way to do that. The framework does have an implementation of this, but, naturally, it is marked internal. Not content to have to deal with writing a non trivial Linq to List<T> provider, I wrote the following:

private static IQueryable<T> CreateQueryable<T>(IEnumerable<T> inner)
{
   Type queryable = typeof (IQueryable).Assembly.GetType("System.Linq.EnumerableQuery`1").MakeGenericType(typeof (T));
   ConstructorInfo constructor = queryable.GetConstructor(BindingFlags.Instance | BindingFlags.NonPublic, 
                    null, new Type[] {typeof (IEnumerable<T>)},new ParameterModifier[0]);
   object instance = FormatterServices.GetSafeUninitializedObject(queryable);
   constructor.Invoke(instance, new object[] { inner });
   return (IQueryable<T>)instance;
}

Not it works.