Ayende @ Rahien

Security decisions: Separate Operations & Queries

Tue, 10 Apr 2012 10:00:00 GMT

The question came up several times in the mailing list with regards to how the RavenDB Authorization Bundle operates, and I think it serves a broader discussion.

Let us imagine a system where we have contracts, which may be in several states:

Mine – Contracts that an employee signed.
Done – Standard users can view, Lawyers assigned to the company can sign.
Draft – Lawyers can view / edit, Partners can approve.
Proposed – Lawyers can create / edit, but only the lawyer that created it can view it, Partners can accept.

So far, fairly simple, right? Except the pure hell that you are going to get into when you are trying to show the users all of the contracts that they can see, sorted by edit date and in the NDA category.

Why am I being so negative here? Well, let us look at what we are going to have to do in the most trivial of cases:

In this sort of system, we are going to have to show the user all of the contracts that they are allowed to see, and show them some indication what operations they can do on each.

The problem is that generating this sort of view is expensive. Especially when you have large amount of data to work through. More interesting, from a UX perspective, it also doesn’t really work that well. Most users would want a better separation of the things that they can do, probably something like this:

This allows us to do a first level filtering on the data itself, rather than try to apply security rules to it.

In the first case, we need to get all the contracts that we are allowed to see. The security rules above are really simple, mind. But trying to translate them into an efficient query is going to be pretty hard. Both in terms of the code requires and the cost to actually perform the query on the server. There are other things that are involved as well, such as paging and sorting in such an environment. I have created several such systems in the past, Rhino Security is probably the most well known of them, and it gets really hard to optimize things and make sure that everything works when you start getting more complex security rules (especially when you have a user editable security system, which is a common request).

The second case is cheaper because we can limit the choices that we see in the query itself. We may still need to apply security concerns, but those goes through the query directly, rather than a security sub system. This kind of change usually force people to be more explicit in what they want, and it result in a system that tends to be simpler. The security rules aren’t just something arbitrary that can be defined, they are actually visible on the screen (My Contracts, Drafts, etc). Changing them isn’t something that is done on an administrator’s whim.

Yes, this is a way to manage the client and their expectations, but that is important. But what about the complex security that they want?

That might still be there, certainly, but that would be active mostly for operations (stuff that happen on a single entity), not on things that happen over all entities. It is drastically easier to make a single entity security decisions work efficiently than make it work over the whole set inside the database.

Monika: A lesson in component based design

Fri, 06 Apr 2012 10:00:00 GMT

I was giving a lecture on architecture recently, and the notion of components came in. The most important bit about that lecture was probably at the very end, when I discussed what it is that I consider to be a component. During that discussion, I introduced Monika, the payment processing component.

Monika has the following Service Level Agreement:

Payment initiation is done messages.
Notification about payment completion is handled via a callback REST call.
The SLA calls for 90% of all successful payments to be processed in 2 business days.

So far, it doesn’t sound really complicated, right? And there isn’t even a hint of how Monika works in the SLA or the contracts.

This is Monica:

Well, not really, but it makes the point, doesn’t it.

Monica is a component in the system that respond to (SMTP) messages, does some work, and respond by clicking on a link in the email (REST call).

Monica has a really sucky SLA, since she has only 22% uptime over the course of the year, and then there are those two weeks when she has her yearly maintenance period (vacation), etc.

The most important thing about this is that we are able to abstract all of that away and treat this scenario as just another component in the system.

All too often, people hear components and they start thinking about things like this:

A component in a system is usually something much larger than a single class or a set of classes. It is an independent agent in the system that has its own behavior, resources, dedicated team and deployment schedule separate from all other components.

Beware the common infrastructure

Thu, 05 Apr 2012 10:00:00 GMT

One of the common problems that I run into when consulting with clients, or just whenever I am talking to developers in general is the notion of common infrastructure. “We are going to spend some time building a common infrastructure which we can then use on all of our applications.”

I made that mistake myself with Rhino Commons, and again very recently with RaccoonBlog (look at the code, you see the Loci stuff, that is stuff that is used from another project).

Why is that a problem? Well, for the simplest reason of all. Different projects have different needs. A common infrastructure that tries to accommodate them all is going to be much more complex. Not only that, it is going to be much more brittle. If I am modifying it in the context of project A, can I really say that I didn’t break something for project B?

Let us take a simple example, executing tasks. In RaccoonBlog, we need tasks merely to handle comments and email (long running background tasks). In another application, we need to do retries, and we need to get notifications if after N retries, the task have failed. In a third project, we need a way to specify dependencies between tasks.

Sure, you can build something that satisfy all three projects, but it would be drastically more complex than having to modify the original task executer for each project needs. And yes, I do mean copying the code and modifying it.

And no, it is not a horrible sin against the Little Endianness. Even duplicated N times, the code is going to be simpler to read, perform faster, easier to maintain and modify over time.

Nitpicker note: I am not talking about 3rd party libraries here. If you can find something that fits your needs that already exists, wonderful. I am talking about infrastructure that you build, inside your organization.

Searching ain’t simple: solution

Fri, 23 Mar 2012 13:06:00 GMT

On my last post, I descried the following problem:

And stated that the following trivial solution is the wrong approach to the problem:

select d.* from Designs d 
 join ArchitectsDesigns da on d.Id = da.DesignId
 join Architects a on da.ArchitectId = a.Id
where a.Name = @name

The most obvious reason is actually that we are thinking too linearly. I intentionally showed the problem statement in terms of UI, not in terms of a document specifying what should be done.

The reason for that is that in many cases, a spec document is making assumptions that the developer should not. When working on a system, I like to have drafts of the screens with rough ideas about what is supposed to happen, and not much more.

In this case, let us consider the problem from the point of view of the user. Searching by the architect name makes sense to the user, that is usually how they think about it.

But does it makes sense from the point of view of the system? We want to provide good user experience, which means that we aren’t just going to provide the user with a text box to plug in some values. For one thing, they would have to put in the architect full name as it is stored in our system. That is going to be a tough call in many cases. Ask any architect what the first name of Gaudi is, and see what sort of response you’ll get.

Another problem is how to deal with misspelling, partial names, and other information. What if we actually have the architect id, and are used to type that? I would much rather type 1831 than Mies Van Der Rohe, and most users that work with the application day in and day out would agree.

From the system perspective, we want to divide the problem into two separate issues, finding the architect and finding the appropriate designs. From a user experience perspective, that means that the text box is going to be an ajax suggest box, and the results would be loaded based on valid id.

Using RavenDB and ASP.Net MVC, we would have the following solution. First, we need to define the search index:

This gives us the ability to search across both name and id easily, and it allows us to do full text searches as well. The next step is the actual querying for architect by name:

Looks complex, doesn’t it? Well, there is certainly a lot of code there, at least.

First, we look for an a matching result in the index. If we find anything, we send just the name and the id of the matching documents to the user. that part is perfectly simple.

The interesting bits happen when we can’t find anything at all. In that case, we ask RavenDB to find us results that might be the things that the user is looking for. It does that by running a string distance algorithm over the data in the database already and providing us with a list of suggestions about what the user might have meant.

We take it one step further. If there is just one suggestion, we assume that this is what the user meant, and just return the results for that value. If there is more than that, we sent an empty result set to the client along with a list of alternatives that they can suggest to the user.

From here, the actual task of getting the designs for this architect becomes as simple as:

And it turns out that when you think about it right, searching is simple.

Searching ain’t simple

Thu, 22 Mar 2012 10:00:00 GMT

The problem statement is best described using:

This seems like a nice and easy problem, right? We join the architects table to the designs table and we are done.

select d.* from Designs d 
 join ArchitectsDesigns da on d.Id = da.DesignId
 join Architects a on da.ArchitectId = a.Id
where a.Name = @name

This is a trivial solution, and shouldn’t take a lot of time to build…

It is also the entirely wrong approach for the problem, can you tell me why?

Composite entities

Fri, 16 Mar 2012 10:00:00 GMT

In my previous post, I discussed some of the problems that you run into when you try to have a single source of truth with regards to an entity definition. The question here is, how do we manage something like a Customer across multiple applications / modules.

For the purpose of discussion, I am going to assume that all of the data is either:

All sitting in the same physical database (common if we are talking about different modules in the same application).
Spread across multiple databases with some data being replicate to all databases (common if we are talking about different applications).

We will focus on the customer entity as an example, and we will deal with billing and help desk modules / applications. There are some things that everyone can agree on with regards to the customer. Most often, a customer has a id, which is shared across the entire system, as well as some descriptive details, such as a name.

But even things that you would expect to be easily agreed upon aren’t really that easy. For example, what about contact information? The person handling billing at a customer is usually different than the person that we contact for help desk inquires. And that is the stuff that we are supposed to agree on. We have much bigger problems when we have to deal with things like customer’ payment status vs. outstanding helpdesk calls this month.

The way to resolve this is to forget about trying to shove everything into a single entity. Or, to be rather more exact, we need to forget about trying to thing about the Customer entity as a single physical thing. Instead, we are going to have the following:

There are several things to note here:

There is no inheritance relationship between the different aspect of a customer.
We don’t give in and try to put what appears to be shared properties (ContactDetails) in the root Customer. Those details have different meaning for each entity.

There are several ways to handle actually storing this information. If we are using a single database, then we will usually have something like:

The advantage of that is that it makes it very easy to actually look at the entire customer entity for debugging purposes. I say for debugging specifically because for production usage, there really isn’t anything that needs to look at the entire thing, every part of the system only care for its own details.

You can easily load the root customer document and your own customer document whenever you need to.

More to the point, because they are different physical things, that solves a lot of the problems that we had with the shared model.

Versioning is not an issue, if billing needs to make a change, they can just go ahead and change things. They don’t need to talk to anyone, because no one else is touching their data.

Concurrency is not an issue, if you make a concurrent modification to billing and help desk, that is not a problem, they are stored into two different locations. That is actually what you want, since it is perfectly all right for having those concurrent changes.

It free us from having to have everyone’s acceptance on any change for everything except on the root document. But as you can probably guess, the amount of information that we put on the root is minimal, precisely to avoid those sort of situations.

This is how we handle things with a shared database, but what is going on when we have multiple applications, with multiple databases?

As you can expect, we are going to have one database which contains all of the definitions of the root Customer (or other entities), and from there we replicate that information to all of the other databases. Why not have them access two databases? Simple, it makes things so much harder. It is easier to have a single database to access to and have replication take care of that.

What about updates in that scenario? Well, updates to the local part is easy, you just do that, but updates to the root customer details have to be handled differently.

The first thing to ask is whatever there really is any need for any of the modules to actually update the root customer details. I can’t see any reason why you would want to do that (billing shouldn’t update the customer name, for example). But even if you have this, the way to handle that is to have a part of the system that is responsible for the root entities database, and have it do the update, from where it will replicate to all of the other databases.

There ain’t no such thing, the definitive entity definition

Thu, 15 Mar 2012 10:00:00 GMT

I was at a customer site, and we were talking about a problem they had with modeling their domain. Actually, we were discussing a proposed solution, a central and definitive definition for all of their entities, so all of the applications could use that.

I had a minor seizure upon hearing that, but after I recovered, I was able to articulate my objections to this approach.

To start with, it breaks the Single Responsibility Principle, the Open Closed Principle and the Interface Segregation Principle. It also makes versioning hard, and introduce a central place where everyone must coordinate with. Think about the number of people that has to be involved whenever you make a change.

Let us take the customer as the representative entity for this discussion. We can all agree that a customer has to have a name, an email and an id. But billing also need to know his credit card information, help desk needs to track what support contracts he has and sales needs to know what sort of products we sold the guy, so we can sell him upgrades.

Now, would you care to be the guy who has to mediate between of all of those different concerns?

And what about changes and updates? Whenever you need to make a change, you have to wait for all of those teams and application to catch up and update and deploy their apps,.

And what about actual usage? You actually don’t want the help desk system to be able to access the billing information, and you most certainly don’t want them to change anything there.

And does it matter if we have concurrent modifications to the entity by both help desk and billing?

All of those things argue very strongly against having a single source of truth about what an entity is. In my next post, I’ll discuss a solution for this problem, Composite Entities.

REST and Urls

Mon, 05 Mar 2012 07:53:00 GMT

Rob Conery has been talking about REST lately, and I think he perpetuate a common misconception. In particular, in the post I referenced, he is asking about ideas for URLs for doing things like logging in, working with productions and episodes, etc.

The problem with that is that this has very little to do with REST. Now, I’ll be the first that will tell you that discussions about architectural purity bore me, and I really like the concept of nice URLs. But nice URLs are totally different from REST.

These slides do a really good work of describing what REST is and how to work with it.

It wasn’t until I actually was called to do a code review on an application written by Rob Eisenberg that I really got it. That application was a pretty simple UI (well, the UI logic was simple, the UI itself was pretty complex, but that was mostly because of the visualizations). The interesting thing is that most of the UI was completely driven by the response from the server.

What I mean by that is that when you loaded an entity, it would load the appropriate view, and use information like this:

<link method="DELETE" title="Cancel" rel="rels/cancelOrder" href="/orders/1234"/>
<link method="GET" title="Shipping Details" rel="rels/viewShipping" href="/orders/1234/shipping"/>

To generate much of the actual behavior on the client side.

The client was fairly stable, but modifying the server meant that you could get a lot more from the system.

Human readable and hackable urls are nice, sure. But they have very little to do with REST.

Do you monitor negative events?

Thu, 23 Feb 2012 10:00:00 GMT

You are probably aware that you need to monitor your production systems for errors, and to add health monitoring for your servers.

But are you monitoring negative events? What is a negative event, stuff that should have happened and didn’t.

For example, every week you have a process that runs to update the tax rates that applies to your customers. This is implemented as a scheduled process, but for some reason (computer was just being rebooted, the user’s password expire, etc) that process didn’t run. There isn’t an error, pre se. You won’t get an error because nothing actually had a chance to actually happen.

Another example would be getting a callback confirmation that an order payment has been correctly processed. That usually happen within 1 – 5 minutes, and you get an OK/Fail notification. But what happens if that notification just never came?

This is a much more dangerous scenario, because you have to not only be prepared for handling errors, you have to be prepared for… nothing to happen.

What it means is that you have to have some way to setup expectations in the system, and act on them when you don’t get a confirmation (negative or positive) within a given time frame.

Limit your abstractions: And how do you handle testing?

Wed, 22 Feb 2012 07:49:00 GMT

On my previous post, I explained about the value of pushing as much as possible to the infrastructure, and then show some code that showed how to do so. First, let us look at the business level code:

[AcceptVerbs(HttpVerbs.Post)]
public ActionResult Register(string originUnlocode, string destinationUnlocode, DateTime arrivalDeadline)
{
    var trackingId = ExecuteCommand(new RegisterCargo
    {
        OriginCode = originUnlocode,
        DestinationCode = destinationUnlocode,
        ArrivalDeadline = arrivalDeadline
    });

    return RedirectToAction(ShowActionName, new RouteValueDictionary(new { trackingId }));
}

public class RegisterCargo : Command<string>
{
    public override void Execute()
    {
        var origin = Session.Load<Location>(OriginCode);
        var destination = Session.Load<Location>(DestinationCode);

        var trackingId = Query(new NextTrackingIdQuery());

        var routeSpecification = new RouteSpecification(origin, destination, ArrivalDeadline);
        var cargo = new Cargo(trackingId, routeSpecification);
        Session.Save(cargo);

        Result = trackingId;
    }

    public string OriginCode { get; set; }

    public string DestinationCode { get; set; }

    public DateTime ArrivalDeadline { get; set; }
}

And the infrastructure code, now:

protected void Default_ExecuteCommand(Command cmd)
{
    cmd.Session = Session;
    cmd.Execute();
}

protected TResult Default_ExecuteCommand<TResult>(Command<TResult> cmd)
{
    ExecuteCommand((Command) cmd);
    return cmd.Result;
}

You might have noticed a problem in the way we are named things, the names on the action and the infrastructure code do not match. What is going on?

Well, the answer is quite simple. Let us look at how our controller looks like ( at least, the important parts ):

public class AbstractController : Controller
{
    public ISession Session;

    public Action<Command> AlternativeExecuteCommand { get; set; }
    public Func<Command, object> AlternativeExecuteCommandWithResult { get; set; }

    public void ExecuteCommand(Command cmd)
    {
        if (AlternativeExecuteCommand!= null)
            AlternativeExecuteCommand(cmd);
        else
            Default_ExecuteCommand(cmd);
    }

    public TResult ExecuteCommand<TResult>(Command<TResult> cmd)
    {
        if (AlternativeExecuteCommandWithResult != null)
            return (TResult)AlternativeExecuteCommandWithResult(cmd);
        return Default_ExecuteCommand(cmd);
    }

    protected void Default_ExecuteCommand(Command cmd)
    {
        cmd.Session = Session;
        cmd.Execute();
    }

    protected TResult Default_ExecuteCommand<TResult>(Command<TResult> cmd)
    {
        ExecuteCommand((Command)cmd);
        return cmd.Result;
    }
}

What?! You do mocking by hand and inject them like that? That is horrible! It is much easier to use a mocking framework and ….

Yes, it would be, if I was trying to mocking different things all the time. But given that I have very few abstractions, it make sense to not only build this sort of infrastructure, but to also build infrastructure for those things _in the tests_.

For example, let us write the test for the action:

[Fact]
public void WillRegisterCargo()
{
  ExecuteAction<CargoAdminController>( c=> c.Register("US", "UK", DateTime.Today) );
  
  Assert.IsType<RegisterCargo>( this.ExecutedCommands[0] );
}

The ExecuteAction method belongs to the test infrastructure, and it setups the controller to be run under the test scenario. Which allows me to not execute the command, but to actually get it.

From there, it is very easy to get to things like:

[Fact]
public void WillCreateNewCargoWithNewTrackingId()
{
  SetupQueryResponse<NextTrackingIdQuery>("abc");
  ExecuteCommand<RegisterCargo>( new RegisterCargo
  {
    OriginCode = "US",
    DestinationCode= "UK",
    ArrivalDeadline = DateTime.Today
  });
  
  var cargo = Session.Load<Cargo>("cargos/1");
  Assert.Equal("abc", cargo.TrackingId);
}

This is important, because now what you are testing is the actual interaction. You don’t care about any of the actual dependencies, we just abstracted them out, but without creating ton of interfaces, abstractions on top of abstractions or any of that.

In fact, we kept the number of abstractions to a minimum, and we can change pretty much every part of the system with very little fear of cascading change.

We have similar lego pieces, all of them move together and interact with one another with complete freedom, and we don’t have to have a Abstract Factory Factory Façade Factory.

Limit your abstractions: The key is in the infrastructure…

Tue, 21 Feb 2012 07:28:00 GMT

In my previous post, I discussed actual refactoring to reduce abstraction, and I showed two very interesting methods, Query() and ExecuteCommand(). Here is the code in question:

[AcceptVerbs(HttpVerbs.Post)]
public ActionResult Register(string originUnlocode, string destinationUnlocode, DateTime arrivalDeadline)
{
    var trackingId = ExecuteCommand(new RegisterCargo
    {
        OriginCode = originUnlocode,
        DestinationCode = destinationUnlocode,
        ArrivalDeadline = arrivalDeadline
    });

    return RedirectToAction(ShowActionName, new RouteValueDictionary(new { trackingId }));
}

public class RegisterCargo : Command<string>
{
    public override void Execute()
    {
        var origin = Session.Load<Location>(OriginCode);
        var destination = Session.Load<Location>(DestinationCode);

        var trackingId = Query(new NextTrackingIdQuery());

        var routeSpecification = new RouteSpecification(origin, destination, ArrivalDeadline);
        var cargo = new Cargo(trackingId, routeSpecification);
        Session.Save(cargo);

        Result = trackingId;
    }

    public string OriginCode { get; set; }

    public string DestinationCode { get; set; }

    public DateTime ArrivalDeadline { get; set; }
}

What are they so important? Mostly because those methods [and similar, like Raise(event) and ExecuteLater(task)] are actually the back bone of the application. They are the infrastructure on top of which everything rests.

Those methods basically accept an argument (and optionally return a value). Their responsibility are:

Setup the given argument so it can run.
Execute it.
Return the result (if there is one).

Here is an example showing how to implement ExecuteCommand:

protected void Default_ExecuteCommand(Command cmd)
{
    cmd.Session = Session;
    cmd.Execute();
}

protected TResult Default_ExecuteCommand<TResult>(Command<TResult> cmd)
{
    ExecuteCommand((Command) cmd);
    return cmd.Result;
}

I have code very much like that in production, because I know that in this system, there are actually only one or two dependencies that a command may want.

There are very few other dependencies, because of the limited number of abstractions that we have. This makes things very simple to write and work with.

Because we abstract away any dependency management, and because we allow only very small number of abstractions, this works very well. The amount of complexity that you have is way down, code reviewing this is very easy, because there isn’t much to review, and it all follows the same structure. The implementation of the rest are pretty much the same thing.

There is just one thing left to discuss, because it kept showing up on the comments for the other posts. How do you handle testing?

Limit your abstractions: Refactoring toward reduced abstractions

Mon, 20 Feb 2012 07:02:00 GMT

So in my previous post I spoke about this code and the complexity behind it:

public class CargoAdminController : BaseController
{
  [AcceptVerbs(HttpVerbs.Post)]
  public ActionResult Register(
      [ModelBinder(typeof (RegistrationCommandBinder))] RegistrationCommand registrationCommand)
  {
      DateTime arrivalDeadlineDateTime = DateTime.ParseExact(registrationCommand.ArrivalDeadline, RegisterDateFormat,
                                                             CultureInfo.InvariantCulture);

      string trackingId = BookingServiceFacade.BookNewCargo(
          registrationCommand.OriginUnlocode, registrationCommand.DestinationUnlocode, arrivalDeadlineDateTime
          );

      return RedirectToAction(ShowActionName, new RouteValueDictionary(new {trackingId}));
  }
}

In this post, I intend to show how we can refactor things. I am going to do that by flattening the architecture, removing useless abstractions and creating a simpler, easier to work with system.

The first thing to do is to refactor the method signature:

[AcceptVerbs(HttpVerbs.Post)]
public ActionResult Register(string originUnlocode, string destinationUnlocode, DateTime arrivalDeadline)

Those are three parameters that we need, there is no need to create a model binder, custom command, etc just for this. For that matter, if you already have a model binder, why on earth do you store the date as a string, and not a date time. The framework is quite happy to do the conversion for me, and if it can’t, I can extend the infrastructure to do so. I don’t need to patch this action with date parsing code.

Next, we have this notion of booking a new cargo, looking at the service, that looks like:

public string BookNewCargo(string origin, string destination, DateTime arrivalDeadline)
{
    try
    {
        TrackingId trackingId = BookingService.BookNewCargo(
            new UnLocode(origin),
            new UnLocode(destination),
            arrivalDeadline
            );
        return trackingId.IdString;
    }
    catch (Exception exception)
    {
        throw new NDDDRemoteBookingException(exception.Message);
    }
}

The error handling alone sets my teeth on edge. Also, note that we have a complex type for TrackingId, which contains just a string (there is a lot of code there for IValueObject<T>, comparison, etc), all of which basically go away if you use an actual string. The same is true for UnLocode (UN Location Code, I assume), but at least this one has some validation code in it.

Then there is the lovely forwarding call, which translate to:

public TrackingId BookNewCargo(UnLocode originUnLocode,
                               UnLocode destinationUnLocode,
                               DateTime arrivalDeadline)
{
    using (var transactionScope = new TransactionScope())
    {
        TrackingId trackingId = cargoRepository.NextTrackingId();
        Location origin = locationRepository.Find(originUnLocode);
        Location destination = locationRepository.Find(destinationUnLocode);

        Cargo cargo = CargoFactory.NewCargo(trackingId, origin, destination, arrivalDeadline);

        cargoRepository.Store(cargo);
        logger.Info("Booked new cargo with tracking id " + cargo.TrackingId);

        transactionScope.Complete();
        return cargo.TrackingId;
    }
}

And now we got somewhere, we actually have something there that is actually meaningful. I’ll skip going deeper, I am pretty sure that you can understand what is going on.

From my point of view of the common abstractions in an application:

Controllers
Views
Entities
Commands
Tasks
Events
Queries

Controllers are at the boundaries of the system, they orchestrate the entire system behavior. Note that I have no place for services or repositories in this list. That is quite intentional. Instead of going that route.

Take a look at the code that I ended up with:

 [AcceptVerbs(HttpVerbs.Post)]
 public ActionResult Register(string originUnlocode, string destinationUnlocode, DateTime arrivalDeadline)
 {
     var origin = Session.Load<Location>(originUnlocode);
     var destination = Session.Load<Location>(destinationUnlocode);

     var trackingId = Query(new NextTrackingIdQuery());

     var routeSpecification = new RouteSpecification(origin, destination, arrivalDeadline);
     var cargo = new Cargo(trackingId, routeSpecification);
     Session.Store(cargo);

     return RedirectToAction(ShowActionName, new RouteValueDictionary(new {trackingId}));
 }

As you can see, the entire architecture was collapsed into a single method.

And what kind of abstractions do we have here?

Well, we have the usual things from MVC, Controller, Action, parameter binding.

We have the session that we are using to load data by id, and to store the newly create cargo.

And we have the notion of a query. Generating a new TrackingID is a query that happen on the database (actually implemented as a hilo sequence). That is something that is definitely not the responsibility of the controller action, so we moved it into a query. Note that we have the Query() method there. It is defined as:

protected TResult Query<TResult>(Query<TResult> query)

And NextTrackingIdQuery is defined as:

public class NextTrackingIdQuery : Query<string>

Pretty simple, overall. And I can hear the nitpickers climb over the fences, waving the pitchforks and torches. “What happen when you need to reuse this logic? It is not in the UI and …”

There are a couple of things to note here.

First, there isn’t anywhere else that needs to book a cargo. And saying “and what happen when…” flies right into a wall of people shouting YAGNI.

Second, let us assume that there is such a need, to reuse the booking cargo scenario. How would we approach this?

Well, we can encapsulate the logic for the controller inside a Command. Which gives us:

[AcceptVerbs(HttpVerbs.Post)]
public ActionResult Register(string originUnlocode, string destinationUnlocode, DateTime arrivalDeadline)
{
    var trackingId = ExecuteCommand(new RegisterCargo
    {
        OriginCode = originUnlocode,
        DestinationCode = destinationUnlocode,
        ArrivalDeadline = arrivalDeadline
    });

    return RedirectToAction(ShowActionName, new RouteValueDictionary(new { trackingId }));
}

And then we have the actual RegisterCargo command:

public abstract class Command
{
    public IDocumentSession Session { get; set; }
    public abstract void Execute();

    protected TResult Query<TResult>(Query<TResult> query);
}
public abstract class Command<T> : Command
{
    public T Result { get; protected set; }
}

public class RegisterCargo : Command<string>
{
    public override void Execute()
    {
        var origin = Session.Load<Location>(OriginCode);
        var destination = Session.Load<Location>(DestinationCode);

        var trackingId = Query(new NextTrackingIdQuery());

        var routeSpecification = new RouteSpecification(origin, destination, ArrivalDeadline);
        var cargo = new Cargo(trackingId, routeSpecification);
        Session.Save(cargo);

        Result = trackingId;
    }

    public string OriginCode { get; set; }

    public string DestinationCode { get; set; }

    public DateTime ArrivalDeadline { get; set; }
}

Note that the Command class also have a way to execute queries, in fact, it is the exact same way that we use when we had the code in the controller. We just moved stuff around, not really made any major change, but we can easily start using the same functionality in another location.

I generally don’t like doing this because most functionality is not reused, it is specific for a particular place and scenario, but I wanted to show how you can lift some part of the code and move it to a different location, otherwise people would complain about the “lack of reuse opportunities”.

On my next post I am going to talk about the Query() and ExecuteCommand() methods, and why they are so important.

Limit your abstractions: So what is the whole big deal about?

Thu, 16 Feb 2012 09:10:00 GMT

When I started out, I pointed out that I truly dislike this type of architecture:

And I said that I much rather an architecture that has a far more limited set of abstractions, and I gave this example:

Controllers
Views
Entities
Commands
Tasks
Events
Queries

That is all nice in theory, but let us talk in practice, shall we? How do we actually write code that actually uses this model?

Let us show some code that uses this type of code, this type, this is from the C# port of the same codebase, available here.

public class CargoAdminController : BaseController
{
  [AcceptVerbs(HttpVerbs.Post)]
  public ActionResult Register(
      [ModelBinder(typeof (RegistrationCommandBinder))] RegistrationCommand registrationCommand)
  {
      DateTime arrivalDeadlineDateTime = DateTime.ParseExact(registrationCommand.ArrivalDeadline, RegisterDateFormat,
                                                             CultureInfo.InvariantCulture);

      string trackingId = BookingServiceFacade.BookNewCargo(
          registrationCommand.OriginUnlocode, registrationCommand.DestinationUnlocode, arrivalDeadlineDateTime
          );

      return RedirectToAction(ShowActionName, new RouteValueDictionary(new {trackingId}));
  }
}

Does this looks good to you? Here is what is actually going on here.

You can click on this link look at what is going in here (and I removed some stuff for clarity’s sake.

This stuff is complex, more to the point, it doesn’t read naturally, it is hard to make a change without modifying a lot of code. This is scary.

We have a lot of abstractions here, services and repositories and facades and what not. (Mind, each of those thing is an independent abstraction, not a common one.)

In my next post, I’ll show how to refactor this to a much saner model.

Limit your abstractions: All cookies looks the same to the cookie cutter

Wed, 15 Feb 2012 07:22:00 GMT

One of the major advantages of limiting the number of abstractions you have is that you end up with a lot less “infrastructure” code. This is in quote because a lot of the time I see this type of code doing things like this:

public class BookingServiceImpl : IBookingService  
{

  public override IList<Itinerary> RequestPossibleRoutesForCargo(TrackingId trackingId)
  {
    Cargo cargo = cargoRepository.Find(trackingId);

    if (cargo == null)
    {
      return new List<Itinerary>();
    }

    return routingService.FetchRoutesForSpecification(cargo.routeSpecification());
  }
  
}

I don’t want to see stuff like that. Instead, I want to be able to go into any piece of code and figure out by what it is what it must be doing. All my code follow fairly similar patterns, and the only differences that I have are actual business differences.

Here is the list of common abstractions that I gave before, this time, I am going to go over each one and explain it.

Controllers – Stand at the edge of the system and manage interaction with the outside world. Can be MVC controllers, MVVM models, WCF Services.
Views - The actual UI logic that is being executed. Can be MVC views, XAML, or real UI code (you know, that old WinForms stuff ).
Entities – Data that is being persisted.
Commands – A packaged command to do something that will execute immediately. (Usually invoked by controllers).
Tasks – A packaged execution that will be execute at a later point in time (usually async), after the current operation have completed.
Events – Something that happened in the system that is interesting and require action. Common place for business logic and interaction.
Queries – Packaged query to be executed immediately. Usually only fairly complex ones gets promoted to an actual query object.

There might be a few others in your system, but for the most part, you would see those types of things over and over and over again.

Oh, sure, you might have other things as well, but those should be rare. If you need to display things in multiple currency interacting with a currency service is something that you would need to often, by all means, make it easy to do (how you do that is usually not important), but the important thing to remember is that those sort of things are one off, and they should remain one off, not the way you structure the entire app.

The reason this is important is that once you have this common infrastructure and shape (for lack of a better word), you can start working in a very rapid pace, without being distracted, and making changes becomes easy. All of your architecture is going through the same central pipes, shifting where they are going is easy to do. You don’t have to drag a rigid system made of a lot of small individual pieces, after all.

Limit your abstractions: Commands vs. Tasks, did you forget the workflow?

Tue, 14 Feb 2012 07:35:00 GMT

On my last post, I outlined the major abstractions that I tend to use in my applications.

Controllers
Views
Entities
~~Commands~~
Tasks
Events
Queries

I also said that I like Tasks much more than Commands and I’ll explain that in the future. When talking about tasks, I usually talk about something that is based on this code. This give us the ability to write code such as this:

public class AssignCargoToRoute : BackgroundTask
{
  public Itinerary Itinerary { get;set; }
  TrackingId TrackingId { get;set; }

  public override void Execute()
  {
    
  }
}

On the face of it, this is a very similar to what we have had before. So why am I so much in favor of tasks rather than commands?

Put simply, the promises that they make are different. A command will execute immediately, this is good when we are encapsulating common or complex piece of logic. We give it a meaningful name and move on with our lives.

The problem is that in many cases, executing immediately is something that we don’t want. Why is that?

Well, what happen if this can take a while? What if this requires touching a remote resource (one that can’t take part of our transaction)? What happen if we want this to execute, but only if the entire operation have been successful? How do we handle errors? What happen when the scenario calls for a complex workflow? Can I partially succeed in what I am doing? Can you have a compensating action if some part fail? All of those scenarios basically boil down to “I don’t want to execute it now, I want the execution to be managed for me”.

Thing about the scenario that we actually have here. We need to assign a cargo to a route. But what does that means? In the trivial example, we do that by updating some data in our local database. But in real world scenario, something like that tends to be much more complex. We need to calculate shipping charges, check manifest, verify that we have all the proper permits, etc. All of that takes time, and usually collaboration with external systems.

For the most part, I find that real world systems requires a lot more tasks than commands. Mostly because it is actually rare to have complex interaction inside your own system. If you do, you have to be cautious that you aren’t adding too much complexity. It is the external interactions that tends to makes life… interesting.

This has implications on how we are building the system, because we don’t assume immediate execution and the temporal coupling that comes with it.

Limit your abstractions: You only get six to a dozen in the entire app

Mon, 13 Feb 2012 10:00:00 GMT

Let us take a look at another part of the DDD sample application. This time, the booking service.

One thing that is really glaring at me is that we have a mix of both commands and queries in this interface. Also, just consider the name. It is a Booking service, but it doesn’t actually seem to have an actual meaning in the application itself. There is no entity named Booking, and except for the BookNewCargo, there is no mention of booking anywhere in the application.

You know what, maybe there is logic in the RequestPossibleRoutesForCargo that is actually meaningful beyond a simple query. Such as charging the customer for the calculation, or reserving space on the route that we selected, etc.

At any rate, I don’t like this service at all. In fact, any time that you have something that is called XyzService, you ought to suspect it. Let me take this one step further. I don’t like that it is an interface, and I don’t like how it is composed. I don’t see it as an independent thing. Going further than that, I don’t really see a reason why we would need an interface here. You might have noticed what the title of this series is. I want to limit abstraction, and IBookingService is an abstraction, one that I don’t see any value in.

In most applications, I like to have a very small number of abstractions. Usually in the order of half a dozen to a dozen (top!). I usually think about them like this:

Controllers
Views
Entities
Commands
Tasks
Events
Queries

Sometimes you have a few more, but those are the major ones. Controllers,Views and Entities are fairly obvious, I would imagine. But what about the rest?

Command is a package “thing” that happen immediately. Tasks are very like commands, except that they don’t have an explicit execution date and Events allow you to build smarts into the system. We have already seen how I handle events, in the previous posts in this series. And Queries should be fairly obvious as well.

Let us tackle Commands now, and then explain why they are bad later on.

When I said that I don’t want abstractions, I meant that I don’t want an interface and an implementation, and some way to connect the two, etc. I don’t really see a lot of value in that for the common case.

Let us break it apart into commands, which would give us this:

public class AssignCargoToRoute : Command
{
  public Itinerary Itinerary { get;set; }
  TrackingId TrackingId { get;set; }

  public override void Execute()
  {
    
  }
}

public class BookNewCargo : Command
{
  public UnLocode Origin {get;set;}
  public UnLocode Destination { get;set; }
  public DateTime ArrivalDeadline {get;set;}

  public TrackingId Result {get;set;}
  public override void Execute()
  {
    
  }
}

I am very fond in having a base class for those types of things. The base class provide me with the infrastructure support for the command in question.

In my next post, I’ll go over why I don’t like this approach, and discuss other ways to structure things so it is more suitable for an actual application.

Limit your abstractions: Application Events–event processing and RX

Fri, 10 Feb 2012 07:55:00 GMT

In my last post, I mentioned that this is actually an event processing system, so we might as well use actual event processing and see what we can gain out of this. I chose to use RX (reactive extensions), which can turn a series of events into a linq statement. This is incredibly powerful, and has some interesting implications when you combine this with your architecture. In particular, let us see what we can get when we set out to replace this with RX based event processing style.

We can get to something like this very easily:

public class CargoProcessor : EventsProcessor
{
    public CargoProcessor()
    {
        On<Cargo>(cargos =>
            from cargo in cargos
            where cargo.Delivery.Misdirected
            select MisdirectedCargo(cargo)
            );

        On<Cargo>(cargos =>
            from cargo in cargos
            where cargo.Delivery.UnloadedAtDestination
            select CaroArrived(cargo)
        );
    }

    private object CaroArrived(Cargo cargo)
    {
        // handle event
        return null;
    }

    private object MisdirectedCargo(Cargo cargo)
    {
        // handle event
        return null;
    }
}

We use RX to handle the linq processing over the events, and in EventsProcessor we have very little code, probably just:

    public class EventsProcessor
    {
        private readonly List<Func<IObservable<object>, IObservable<object>>>  actions = new List<Func<IObservable<object>, IObservable<object>>>();

        protected void On<T>(Func<IObservable<T>, IObservable<object>> action)
        {
            actions.Add(observable => action(observable.OfType<T>()));
        }

        public void Execute(IObservable<object> observable)
        {
            foreach (var action in actions)
            {
                action(observable).Subscribe();
            }
        }
    }

Elsewhere in the code we setup the actual Obsersable that we pass to all the EventsProcessors. The major advantages that we have with this style is that we have a natural syntax to do selection on the events that interest us, including fairly complex one. We still have easy time of creating new EventsProcessors if we want, but because the code for defining the selection is so compact, we can usually put related stuff together, which is going to be very helpful for making sure that the codebase is readable.

And, naturally, this method extends itself to handling events of multiple types in the same place. For example, if we want to also handle the HandlingEvent, we can do it in place, because it is very much related to the Cargo, it seems.

Limit your abstractions: Application Events–Proposed Solution #2–Cohesion

Thu, 09 Feb 2012 00:41:00 GMT

In my previous post, I spoke about ISP and how we can replace the following code with something that is easier to follow:

I proposed something like:

public interface IHappenOn<T>
{
   void Inspect(T item);
}

Which would be invoked using:

container.ExecuteAll<IHappenOn<Cargo>>(i=>i.Inspect(cargo));

Or something like that.

Which lead us to the following code:

public class CargoArrived : IHappenedOn<Cargo>
{
  public void Inspect(Cargo cargo)
  {
    if(cargo.Delivery.UnloadedAtDestination == false)
      return;
      
    // handle event
  }
}

public class CargoMisdirected : IHappenedOn<Cargo>
{
  public void Inspect(Cargo cargo)
  {
    if(cargo.Delivery.Misdirected == false)
      return;
      
    // handle event
  }
}

public class CargoHandled : IHappenOn<HandlingEvent>
{
   // etc
}

public class EventRegistrationAttempt : IHappenedOn<HandlingEventRegistrationAttempt>
{
  // etc
}

But I don’t really like this code, to be perfectly frank. It seems to me like there isn’t really a good reason why CargoArrived and CargoMisdirected are located in different classes. It is likely that there is going to be a lot of commonalities between the different types of handling events on cargo. We might as well merge them together for now, giving us:

public class CargoHappened : IHappenedOn<Cargo>
{
  public void Inspect(Cargo cargo)
  {
    if(cargo.Delivery.UnloadedAtDestination)
      CargoArrived(cargo);
      
    
    if(cargo.Delivery.Misdirected)
      CargoMisdirected(cargo);
      
  }
  
  public void CargoArrived(Cargo cargo)
  {
    // handle event
  }
  
  public void CargoMisdirected(Cargo cargo)
  {
    //handle event
  }
}

This code put a lot of the cargo handling in one place, making it easier to follow and understand. At the same time, the architecture gives us the option to split it to different classes at any time. We aren’t going to end up with a God class for Cargo handling. But as long as it make sense, we can keep them together.

I like this style of event processing, but we can probably do better job at if if we actually used event processing semantics here. I’ll discuss that in my next post.

Limiting your abstractions: Reflections on the Interface Segregation Principle

Wed, 08 Feb 2012 07:35:00 GMT

I have found two definitions for ISP (in this post, I assume that you know what ISP is, if you don’t, look at the links below).

This one I can fully agree with:

Clients should not be forced to depend upon interfaces that they don't use.

And this one I strongly disagree with:

Many client specific interfaces are better than one general purpose interface

For fun, both of those links are from Object Mentor.

It would be more accurate to say that I don’t so much disagree with the intent of the second statement, but that I disagree with the wording. One (literal) way of understanding the second statement is to say that we want specific interfaces over generic ones. And I would strongly disagree. An interface is an abstraction (for the most part), and I want to reduce the amount that I have in my system.

Let us look at an example, based on my recent posts in this series.

In my previous post, I said that I want to remove both CargoWasHandled and CargoWasMisdirected in favor of IHappenOnCargoInspection. But what about the two other methods?

Would ISP force me to have IHappenOnEventRegistrationAttempt and IHappenOnCargoHandling?

Sure, those are many specific interfaces, but are they really better than something like IHappenOn<T>? Is there something truly meaningful that gets lost when we have the generic interface?

I would say that this isn’t the case, further more, I would actually state that having a single interface will lead to more maintainable code, because there are less abstractions in the code. There is a codebase that is more cohesive and easier to understand.

But I’ll talk more about cohesion on my next post.

Limit your abstractions: Application Events–Proposed Solution #1

Tue, 07 Feb 2012 07:30:00 GMT

In my previous post, I explained why I really don’t the following.

public class CargoInspectionServiceImpl : ICargoInspectionService 
{
  // code redacted for simplicity

 public override void InspectCargo(TrackingId trackingId)
 {
    Validate.NotNull(trackingId, "Tracking ID is required");

    Cargo cargo = cargoRepository.Find(trackingId);
    if (cargo == null)
    {
      logger.Warn("Can't inspect non-existing cargo " + trackingId);
      return;
    }

    HandlingHistory handlingHistory = handlingEventRepository.LookupHandlingHistoryOfCargo(trackingId);

    cargo.DeriveDeliveryProgress(handlingHistory);

    if (cargo.Delivery.Misdirected)
    {
      applicationEvents.CargoWasMisdirected(cargo);
    }

    if (cargo.Delivery.UnloadedAtDestination)
    {
      applicationEvents.CargoHasArrived(cargo);
    }

    cargoRepository.Store(cargo);
 }
}

Now, let us see one proposed solution for that. We can drop the IApplicationEvents.CargoHasArrived and IApplicationEvents.CargoWasMisdirected, instead creating the following:

public interface IHappenOnCargoInspection
{
   void Inspect(Cargo cargo);
}

We can have multiple implementations of this interface, such as this one:

public class MidirectedCargo : IHappenOnCargoInspection
{
   public void Inspect(Cargo cargo)
   {
        if(cargo.Delivery.Misdirected == false)
             return;

        // code to handle misdirected cargo.
   }
}

In a similar fashion, we would have a CargoArrived implementation, and the ICargoInspectionService would be tasked with managing the implementation of IHappenOnCargoInspection, probably through a container. Although I would probably replace it with something like:

container.ExecuteOnAll<IHappenOnCargoInspection>(i=>i.Inspect(cargo));

All in all, it is a simple method, but it means that now the responsibility to detect and act is centralized in each cargo inspector implementation. If the detection of misdirected cargo is changed, we know that there is just one place to make that change. If we need a new behavior, for example, for late cargo, we can do that by introducing a new class, which implement the interface. That gives us the Open Closed Principle.

This is better, but I still don’t like it. There are better methods than that, but we will discuss them in another post.