Design

Patterns vs. Blueprints

Ayende Rahien — Sat, 12 Apr 2008 01:30:03 GMT

Recently I had several discussions about the usage of patterns, and application thereof. Usually, there are recommended implementation for a pattern, often this is the way it is presented in its pattern catalog.

The discussion had to do with the amount of fidelity that you had in implementing the pattern. My view on that is that the usage of a pattern doesn't require a one to one match with the common implementation, but leave room for change while keeping the concepts of the pattern in use.

Patterns are the overall shape of the solution, they are not a blueprint, to be slavishly followed.

Request/Reply vs. Pub/Sub

Ayende Rahien — Thu, 10 Apr 2008 20:52:37 GMT

I am having an interesting discussion in email with Evan Hoff, about the benefits of pub/sub architecture vs. request/reply. I want to state upfront that I see a lot of value in async one way messaging for a lot of the interactions in an application and across services. However, I am still skeptic about the push for pub/sub all the way.

To me, it seems like the request/reply model is a very natural one in many common scenarios. In most backend processing systems, I would tend to use pub/sub with sagas as both the easiest and most scalable solution. But in almost all cases that end up in the UI, this just doesn't make sense. Even if I want to fire several requests and then wait for all their reply, the model is still more natural than a pub/sub approach.

Note that the following code sample use imaginary API.

Let us go back to the displaying the available candidates. Using the request/reply model, I would have:

public void AvailableCandidates()

{
     var specifications = BuildSpecifications();
     var msg = bus.Process(new FindAvailableCandidateMessage(specifications));
     DisplayResults(msg.Results);
}

All of those are using the request / reply model.

The pub/sub model will look somewhat like this:

public void AvailableCandidates()
{
	var specifications = BuildSpecification();
	
	bus.Publish(new FindAvailableCandidatesMessage(specifications))
		.WaitForCallback(delegate(AvailableCandidatesMessage msg)
		{
			DisplayResults(msg.Results); 
		});
	
}

Or probably something like:

public void AvailableCandidates()
{
	var specifications = BuildSpecification();
	Guid correlationId = Guid.NewGuid();
	ManualResetEvent waitForReply = new ManualResetEvent(false);
	
	bus.SubscribeOnce<AvailableCandidatesMessage>(correlationId, delegate(AvailableCandidatesMessage msg)
		{
			DisplayResults(msg.Results); 
			waitForReply.Set();
		});
	bus.Publish(new FindAvailableCandidatesMessage(correlationId, specifications));
	waitForReply.WaitOne();
}

Notice that I am not saying anything about the way those are handled once they leave the client. In fact, some of the reasons that were mentioned to favor the pub/sub model is the flexibility that you get when adding more functionality. Such as adding auditing, or redirecting messages to different services based on the type of candidate, etc.

The problem is that I see not inherent need for pub/sub to make this work. In fact, I think that anything that publish a message and than waits for the callback message is a big API issue, because this is request/reply, but with harder to use API. Even if you are doing the wait in async manner, I still think that this is a bad idea.

In the context of a request/reply interaction, we can handle everything the way we would handle any other message:

In fact, from the perspective of the infrastructure, there is little need to distinguish those. A message going out will go out either via a request/reply channel or a one way output channel, with a reply maybe arrived through a one way input channel. We don't really care about that.

From my point of view, it makes a lot more sense than to wait for a callback.

Thoughts?

SOA Data Access

Ayende Rahien — Wed, 09 Apr 2008 23:30:30 GMT

Yesterday I posted about designing an SOA system. I identified the following service:

Available candidates - given a set of requirements, will attempt to find all verified candidates that match the requirements. Additional responsibilities include logging search history (important for noticing what requirements are common and desirable), and recording which requirements has no matching physicians in the system (should be passed for human inspection, to decide if head hunting should begin, or this should be ignored).

BIll Poole has left a comment regarding this post, which included this statement:

For example, the "Available Candidates" service appears highly data centric, likely exposing search/data retrieval operations. CRUD interfaces are bad.

This is something that I have often heard, but I still don't understand. I understand why CRUD interfaces are discourage, you get better results by just using a DB. But in this case?

How do you get the data out of an SOA system? It seems to me like this is a natural requirement for the problem at hand, but I feel that I am missing something.

An exercise in designing SOA systems

Ayende Rahien — Tue, 08 Apr 2008 05:39:53 GMT

This question came up in the NServiceBus mailing list, and I think it is very interesting question to try to solve. The standard disclaimers are that I haven't build a real system using this approach, although I am leaning toward it more and more.

I would like to thank Brad for agreeing to post this example here.

I would like some advice as to how to properly identify the services I need to create. I know that Udi has said that creating services for each department is a good starting point, but I keep feeling like something is missing.

So, here's some details. I work for a company that is the parent of two physician job placement companies. The parent company would like to create a single system that will support both child companies. The business processes of both companies have been analyzed and documented, and there is about a 95% overlap in the processes. Should my team create a monolithic job placement application and wrap it in a service or should I break that application into higher-level business services?

Here's an example of where I get stuck when trying to map the job placement application's functionality to a high-level business service like Sales Service. The Sales department is broken into two groups: marketers and recruiters. Marketers call hospitals to find out what positions are available and try to fill these positions. Recruiters call physicians to find out if the physician is looking for a new job and tries to find available positions. Before a physician can be placed, his or her credentials must be verified. The Credentialing department is responsible for doing this.

Ok. So I now have two service candidates (Sales and Credentialing). I have no issues with the credentialing service but the sales service feels like it should be a job placement service. What if the business decides to take on a new line of business? This new line-of-business may have the sales department selling something entirely unrelated to job placement services. Is a job placement service a better idea? But wouldn't that mean that all job placement functionality would need to be placed in that service? This feels like I'm loosing the agility that SOA gives the business because I'm no longer modeling the way the departments interact with each other.

I once built a system with very similar concepts, although with drastically different focus. What Brad has already outlined is a good start, but I would break the system even further. We have two main sections of the business that were identified, marketing and recruitment. Let us attack the problem from each side in turn, but keep in mind that we are talking about a single system here.

High level services:

Available candidates - given a set of requirements, will attempt to find all verified candidates that match the requirements. Additional responsibilities include logging search history (important for noticing what requirements are common and desirable), and recording which requirements has no matching physicians in the system (should be passed for human inspection, to decide if head hunting should begin, or this should be ignored).
Register open position - enter a new position into the system, with all its requirements. As part of the registration, it will query the available physicians and offer them as matches. If there are no matches, it will register the new position in the recruitment service.
Register new candidate - enter a new candidate to the system in unverified state. Request credentials verifications from the system.
Verify credentials - given a candidate, perform a verification of the candidate credentials, references, etc. If the verification succeeds, we move the candidate to the available candidate pool. If the verification fails, we request additional intervention.
Problematic credentials resolver - there are several reasons for getting problematic credentials, fake credentials, honest mistakes and hostile employers are just a few that comes to mind. In most cases, you need complex logic or human intervention here, and it deserves its own section in the map.
Candidate hunger* - which candidates are we looking for, this is used by the recruitment guys to know which candidates they should aim for.
Candidate tracker - if we already assigned a candidate for a job, we already have a happy person, let us keep it this way by ensuring that they remember us, so when they leave, they will contact us again for their next job.
Potential candidates - given a set of requirements, will attempt to find all candidates that match them. Potential candidates are usually candidates that were gotten from external data sources (the physician listing for the country, for instance), so you want to go through them to see if they can become real candidates.

Those are just the services that I can think of off the top of my head. Notice that each of them has very few responsibilities, and they match pretty closely a specific business scenario. The services communicate between one another, but they do so using one way, async messaging. This is especially important in this type of scenario, where some services can take long time to process a piece of work (verifying credentials requires a human to call and ask for references, for example).

My next post will deal with the actual code for one of those services, for now I think this is enough.

* It was originally a typo, but I like the implications.

Code Generation and the Open Close Principal

Ayende Rahien — Sat, 29 Mar 2008 01:26:00 GMT

Vladan Strigo made a really inspiring comment about code generation in the NH Users list.

I didn't want to use codesmith for that because it would de-OCP-fy me in my future efforts :)

That makes a lot of sense, and manages to touch on what bothers me the most with code gen as an architectural approach. It gives up a very important concept, and that affect a lot of the stuff that is dependant on that.

Non Invasive API, take 2

Ayende Rahien — Mon, 24 Mar 2008 22:58:25 GMT

A few days ago I presented my typical way of handling container driven polymorphism.

// only use for selection
public interface IRegion<T> : IRegion {}

public void SetRegion(DependencyObject containerElement, string regionName)
{
	IRegion region = (IRegion) IoC.TryResolve(typeof(IRegion<>).MakeGenericType(containerElement.GetType()));

	if (region != null)
		_regions.Add(regionName, region);
}

I got an email with some good questions, and it is worth a second post to clarify. The questions:

The only thing I don’t like is that a lookup in the form of TryResolve(typeof(IRegion<>).MakeGenericType(containerElement.GetType())) would infer an explicit wrapper type for ever possible UI container type. On one hand it gives me the fine grain control I need to specify a different wrapper for an inherited UI container. But on the other hand, it means that ListBox, TreeView, ComboBox and TabControl would all need their own custom wrappers even though they all derive from ItemsControl and all handle adding and removing child entities in the same way.

But if containerElement == typeof(ListBox), won’t TryResolve(typeof(IRegion<>).MakeGenericType(containerElement.GetType())) fail to resolve? TryResolve expects an explicit match. It would need to find a registration for IRegion<ListBox>, not IRegion<ItemContainer> right? (Assuming Unity here.)

Even if that worked, what if I want all elements that inherit from ItemContainer to be handled by IRegion<ItemContainer> except TreeView? Say I need to handle TreeView differently for some reason. What’s a nice way to handle the 80% rule generically but still allow an override for special cases?

My response to that is very simple. Take the same approach, but bigger. Here is all we need to make this happen.

public void SetRegion(DependencyObject containerElement, string regionName)
{
	Type current = containerElement.GetType()
	while(current != typeof(DependencyObject))
	{
		IRegion region = (IRegion) IoC.TryResolve(typeof(IRegion<>).MakeGenericType(current));
	
		if (region != null)
		{
			_regions.Add(regionName, region);
			return;
		}
		
		current = current.BaseType;
	}
}

Now, we can register the following:

IRegion<TreeView>
IRegion<MyCustomRegion>
IRegion<ItemsControl>

And no matter what we will send to SetRegion, we can modify the behavior merely by changing the registration of IRegion<T> implementations. Note that this doesn't assumes any capacity on the container (except the basic resolve-generic-service one).

This is also a very natural model to follow, in nearly all cases (there is an issue here regarding whatever we should consider interfaces as well, but that is beside the point for this demo, and I would generally not do that anyway).

You should hard code the idea, not the value

Ayende Rahien — Sun, 23 Mar 2008 19:51:13 GMT

I am reading NServiceBus' code at the moment, and I run into this idea:

[Serializable]
[Recoverable]
[TimeToBeReceived(“0:01:00.000")]
public class ScheduleInterviewMessage : IMessage
{
    public Guid InterviewerId;
    public Guid CandidateId;
    public DateTime RequestedTime;
}

I don't like hard coding this value in the message. Obviously this is something that is tied very closed to the idea of the message. Time to live is certainly something that I would want to closely tie to the message itself. But hard coding the value goes against a lot of instincts.

A while ago I faced a similar issue, in that case, it was scheduling tasks. The timing of the task was tied to what it did. The last example that I had on that post is this:

[Occurances(OccureEvery.Day, At = "08:00", RunAtHolidays = false)]
public class SendSmsTwoDaysBeforeEventStart : ScheduledTask

Notice the At = "08:00" ? It bothered me for a while, until I came to the conclusion that I was embedding the wrong piece of information in the code.

I didn't want this to run at 08:00, I want it to run in the morning. 08:00 was just a handy designation for that. That thought was quickly followed by:

public enum ScheduledTimes
{
     Midnight,
     EarlyMorning,
     Morning,
     Noon,
     Afternoon,
     Evening
}

And the code turned to:

[Occurances(OccureEvery.Day, At = ScheduledTimes.Morning, RunAtHolidays = false)]
public class SendSmsTwoDaysBeforeEventStart : ScheduledTask

The difference here is that I no longer hard code a specific value, I put the idea there. During deployment, I could have modified what morning meant, very easily.

Using the same approach, I would define something like:

public enum TimeToLive
{
    Immediate,
    Short,
    Medium,
    Long,
    Forever
}

I would probably allow to configure the time that each of those values map to, as well.

This allows to do things like update them on the fly (imagine having to survive a burst of traffic that slows the entire system), so you can increase the time to process a message without losing it.

A maintainable environment - anti corruption as a way of life

Ayende Rahien — Sat, 22 Mar 2008 16:26:16 GMT

Recently I had a number of conversations about how to create a maintainable environment, not only for advanced developers, but an environment that enables beginners and intermediate developers to create good, maintainable, scalable solutions.

This problem is not a simple one, and there aren't really one off solutions. However, there is quite a lot that can be done, and I am very disturbed by the assumption that nothing can be done about it.

I noticed that a major part of the problem seems to be in the underlying assumption that in order to make it possible for beginners/intermediate developers to work with the application, you have to do one of two things. Embark on a long education period or dumb down the application to a level that beginners can deal with.

I am using the term "dumb down" on purpose. This approach goes for the easiest solution possible, those solutions tend to be very easy and supportive for trivial scenarios, but they get very complicate for any real world scenario. They also tend to be focused on RAD, with little regard to the overall maintainability of the solution. I am pretty sure that you have encounter such systems before, they are depressingly common.

The end result of choosing such an approach is an application whose complexity is not handled by the underlying platform, but rather at each individual case. This significantly complicate the applicatIon and working with in over time. As a quick example (which is sure to annoy a lot of people), I consider any application that has "new SqlConnection()" in it outside the infrastructure layer to be such a system.

The other alternative, however, is a lengthy training cycle, and slower productivity all around until you get to be proficient in the new approach. And, of course, some people will have really hard time adjusting.

Or so many people believe, at least. I strongly disagree with the previous statement.

The way I would choose for such a scenario (a team composed of developers of varying skills, being non trivial application) is to accept the fact that you cannot escape from the complexity and embrace that fact. Instead of trying to base the application on an overly simple foundation, I would base it on advance platform.

This advance platform would be able to scale to handle the complexity of the application, while concentrating that complexity into the infrastructure layer.

Some of the concepts involved in creating such an architecture are:

Infrastructure Ignorance - If I have to deal with my infrastructure explicitly, then I need to have developers dealing with it on a regular basis. That doesn't mesh well with the idea of moving all the complexity to the infrastructure.
Fizz Buzz concepts - the code that a developer should write should be simple and easy to follow.
Follow good design principals - the code written should naturally lean toward single responsibility principal, separation of concerns, etc
Easy to figure out - It should be simpler than the alternative for any non trivial scenario. Ideally, you can explain how to solve most scenarios using this approach in half an hour or less (ignoring the infrastructure part, which may be very complex).
No training wheels - one of the things that drives me crazy is having two ways to do things. One for the average developer, the second for the architect. This usually comes out of putting training wheels on the first approach that make it hard to deal with in more advance scenarios. I see this as a sign of contempt for the rest of the members of the team. If you expect the rest of the team to work in a certain way, make damn sure that this is a way that this is a way that you can live with.

I think that I manage to get fairly close to this in my previous project, and it has proven to be a successful one. I talk about it a bit in this post.

I based that system on IoC, NHibernate and WCF. None of this concepts was exposed to the other developers. In fact, after setting this up, we were able to focus on the actual business issues far more often than on the technology involved.

By the way, I would fully expect that during the project, significant attempts would be made to get the other developers to understand. This technique is about getting a project started and the team productive very early, and doing this using a stable foundation.

Now, what is this anti corruption stuff in the title of this post? This is very simple, actually. It is here as more than the anti corruption than the anti corruption layers that Feathers is talking about in Working Effectively With Legacy Code. In this case, we start with the assumption that the infrastructure that we will use (even if we are the one building it) is suspect and should not be trusted, so we will do everything in our power to isolate our code from the infrastructure.

An important benefit of this approach is that we get the ability to modify the infrastructure independently of our application. This means that we can make significant changes very easily, because there is little to no dependencies between the two.

Non invasive API - Now with an IoC container

Ayende Rahien — Wed, 19 Mar 2008 21:37:41 GMT

I talked about ways to avoid invasive API design, and a lot of people asked about how to handle this with a container. First, I want to make it clear that the previous example assumed that you can't rely on a container, so it used Poor Man IoC to do that. Now that we can assume that there is a container, this is another matter entirely.

The API design now shifts to allow me to select the proper implementation from the container automatically. This generally ends up being either as a naming convention on top of a fixed interface, or as a generic interface with a given type as a parameter. The decision depends on what you are doing, basically, and the capabilities of your tools.

For myself, I strongly favor the generic interface approach, which would give us the following syntax:

public interface IMessageHandler<TMsg>
	where TMsg : IMessage
{
	void Handle(TMsg msg);
}

public class EndPoint
{
	public void HandleMsg(IMessage msg)
	{
		// this should be cached and turned into a delegate
		// not reflection call
		GetGenericHandleMsgMethod().MakeGenericMethod(msg.GetType())
			.Invoke(this, new object[]{msg});
	}

	public void HandleMsg<TMsg>(TMsg msg)
		where TMsg : IMessage
	{
		IoC.Resolve<IMessageHandler<TMsg>().Handle(msg);
	}

}

There is some ugliness in invoking the method with the generic parameter, but this allows you to handle a wide variety of cases very easily.

Let us take another example, this time it is from the Prism.Services.RegionManagerService:

public void SetRegion(DependencyObject containerElement, string regionName)
{
	IRegion region = null;

	if (containerElement is Panel)
	{
		region = new PanelRegion((Panel)containerElement);
	}
	else if (containerElement is ItemsControl)
	{
		region = new ItemsControlRegion((ItemsControl)containerElement);
	}

	if (region != null)
		_regions.Add(regionName, region);
}

I don't really like this code, let us see what happens when we introduce the idea of the container as a deeply rooted concept:

// only use for selection
public interface IRegion<T> : IRegion {}

public void SetRegion(DependencyObject containerElement, string regionName)
{
	IRegion region = (IRegion) IoC.TryResolve(typeof(IRegion<>).MakeGenericType(containerElement.GetType()));

	if (region != null)
		_regions.Add(regionName, region);
}

Now the code is much clearer, we can extend it from the outside, without modifying anything when we add a new region type.

Avoiding Invasive API design

Ayende Rahien — Wed, 19 Mar 2008 06:45:43 GMT

I have a strong dislike for invasive API and framework. I dislike them because they require a lot of work, avoid existing options, limit my ability to work with them, etc. They also tend to tie you to a platform for very little reason.

Here is a simple (and fictional) example. Let us assume that we want to write a library to expose our domain model in a version way. So we can do something like this:

svn://localhost/Customers/15[@revision=3]

Now, in order to handle this scenario, I have devised the following schema. I can create the end point using the following approach:

public class Customers : Versioned<Customer> { }

And I can can specify the revision using an attribute:

public class Customer
{
	public virtual int Id { get; set; }

	[SvnVersion]
	public virtual int Version { get; set; }
}

Very simple, right? And very straightforward. Except that this in incredibly invasive approach to take.

I really dislike API that work this way. Now, if I want to expose my domain model using SVN, I need to go into the domain model and start messing with that, just because I want to add another accessibility option?

This is bad. It violates the open close principal, single responsibility principal and in Ayende Annoyance Principal.

Let us try to make this easier, shall we?

public class Customers : Versioned<Customer, CustomerVersionPropertyProvider>
{
}

Now I don't have to modify the Customer entity, I can do this in a way that doesn't modify Customer, I keep things separate and in general I am happy. From the Versioned class perspective, I now define it as:

public class Versioned<TEntity, TVersionProvider>
	where TVersionProvider : IVersionProvider, new()
{
}

Actually, this is not really good. We probably want to control the creation of the version provider ourselves (it may need external dependencies, we may want to control its lifetime, etc), so we get to this design:

public class Versioned<TEntity, TVersionProviderFactory>
 where TVersionProviderFactory: IVersionProviderFactory, new() 
{
}

public interface IVersionProviderFactory
{
	IVersionProvider Create();
}

public class Customers : Versioned<Customers, CustomersVerionProviderFactory>
{
}

And now we really have non invasive API, which plays well with other tools.

Except, the simple scenario that we had before turn out to be really complex. Instead of putting a simple attribute in place, we now need to implement two interfaces, do it per class, and in generally add a lot more work into our life.

That is also bad. But we can resolve that easily, by defaulting to the invasive behavior (which is really easy to explain), yet allowing extensibility. The way to do it is simple:

public class Versioned<TEntity, TVersionProviderFactory>
	where TVersionProviderFactory: IVersionProviderFactory, new() 
{
} 

public class Versioned<TEntity> : Versioned<TEntity, SvnVersionAttributeVersionProviderFactory>
{
} 


public class Customers : Versioned<Customer>
{
}

Now we have the same API for the simple (and invasive) scenario, but with minimal effort, we have made it possible to use non invasive and smart approaches.