Recently I updated several of my VS plugins, and immediately after that I noticed a changed in the way VS behaves.

This is actually quite nice, and useful. (Although I was a bit confused for a time about when it shows and when it doesn’t, but that is beside the point).

the problem is that in many cases, like the first two example, it is actually quite easy to press on those notifications. And if you do that, you get:

And that is annoying. Sure, I get that you want to encourage people to buy your products, I even agree. But this sort of a feature is something that is very easy to invoke by mistake, and it completely throws you out of your current context.

I have VSCommands installed for the simple reason that I really like the Reload All feature. But it isn’t worth it if I have to be careful where I put my mouse in VS.

This single feature is the reason that I uninstalled it.

Take a look at the following code:

public static Task ParseAsync(IPartialDataAccess source, IPartialDataAccess seed, Stream output, IEnumerable<RdcNeed> needList)
{
    return Task.Factory.StartNew(() =>
    {
        foreach (var item in needList)
        {
            switch (item.BlockType)
            {
                case RdcNeedType.Source:
                    source.CopyToAsync(output, Convert.ToInt64(item.FileOffset), Convert.ToInt64(item.BlockLength)).Wait();
                    break;
                case RdcNeedType.Seed:
                    seed.CopyToAsync(output, Convert.ToInt64(item.FileOffset), Convert.ToInt64(item.BlockLength)).Wait();
                    break;
                default:
                    throw new NotSupportedException();
            }
        }
    });
}

Do you see the problem in here?

It is a result of a code review comment about improper use of async in a project. This resulted in a lot of Task showing up in the return methods, but not in any measurable improvement in the actual codebase use of asynchronicity.

The problem is that when you need to work with such things in C# 4.0, you have to do some annoying things to get the code to work properly. In particular, this method was modified to be:

public static Task ParseAsync(IPartialDataAccess source, IPartialDataAccess seed, Stream output, IList<RdcNeed> needList, int position = 0)
{
  if(position>= needList.Count)
  {
        return new CompletedTask();
  }
  var item = needList[position];
  Task task;
            
  switch (item.BlockType)
  {
        case RdcNeedType.Source:
            task = source.CopyToAsync(output, Convert.ToInt64(item.FileOffset), Convert.ToInt64(item.BlockLength));
            break;
        case RdcNeedType.Seed:
            task = seed.CopyToAsync(output, Convert.ToInt64(item.FileOffset), Convert.ToInt64(item.BlockLength));
            break;
        default:
            throw new NotSupportedException();
  }

  return task.ContinueWith(resultTask =>
    {
        if (resultTask.Status == TaskStatus.Faulted)
            resultTask.Wait(); // throws
        return ParseAsync(source, seed, output, needList, position + 1);
    }).Unwrap();
}

This code is more complex, but it is actually making proper use of the TPL. We have changed the loop into a recursive function, so we can take advantage of ContinueWith to the next iteration of the loop.

And no, I can’t wait to get to C# 5.0 and have proper await work.

This post is partly in response for this post, discussing the Azure problem with leap year. But it is actually a bit more general than that.

In my code, here is how I define “one year from now”:

DateTime.Today.AddYears(1).AddDays(3);

As it turned out, this tend to have a lot of implications on your business, most of them are actually pretty good ones.

For a start, you will never get hit with a leap year bug, but more importantly, you are never going to have to deal with an immediate cutoff. This is important because it gives you time. Mostly, it gives you time to screw up, but having the time to do so without having an egg all of your face is a really nice thing.

For example, all of our subscriptions are using a similar method of calculation, and this is why we can take the ordering system down for a few hours or even a day or two and no one will actually notice. We have a big grace period in which we can work things out.

Sure, a user gets 3 “extra” days for free out of this, but frankly, I don’t give a damn. It is more important that I get the buffer, and most users like it much better when you don’t slam the doors in their faces on the first chance.

One of the things that is important is style, and giving a grace period for those sort of things is crucial.

The question came up several times in the mailing list with regards to how the RavenDB Authorization Bundle operates, and I think it serves a broader discussion.

Let us imagine a system where we have contracts, which may be in several states:

• Mine – Contracts that an employee signed.
• Done – Standard users can view, Lawyers assigned to the company can sign.
• Draft – Lawyers can view / edit, Partners can approve.
• Proposed – Lawyers can create / edit, but only the lawyer that created it can view it, Partners can accept.

So far, fairly simple, right? Except the pure hell that you are going to get into when you are trying to show the users all of the contracts that they can see, sorted by edit date and in the NDA category.

Why am I being so negative here? Well, let us look at what we are going to have to do in the most trivial of cases:

In this sort of system, we are going to have to show the user all of the contracts that they are allowed to see, and show them some indication what operations they can do on each.

The problem is that generating this sort of view is expensive. Especially when you have large amount of data to work through. More interesting, from a UX perspective, it also doesn’t really work that well. Most users would want a better separation of the things that they can do, probably something like this:

This allows us to do a first level filtering on the data itself, rather than try to apply security rules to it.

In the first case, we need to get all the contracts that we are allowed to see. The security rules above are really simple, mind. But trying to translate them into an efficient query is going to be pretty hard. Both in terms of the code requires and the cost to actually perform the query on the server. There are other things that are involved as well, such as paging and sorting in such an environment.  I have created several such systems in the past, Rhino Security is probably the most well known of them, and it gets really hard to optimize things and make sure that everything works when you start getting more complex security rules (especially when you have a user editable security system, which is a common request).

The second case is cheaper because we can limit the choices that we see in the query itself. We may still need to apply security concerns, but those goes through the query directly, rather than a security sub system. This kind of change usually force people to be more explicit in what they want, and it result in a system that tends to be simpler. The security rules aren’t just something arbitrary that can be defined, they are actually visible on the screen (My Contracts, Drafts, etc). Changing them isn’t something that is done on an administrator’s whim.

Yes, this is a way to manage the client and their expectations, but that is important. But what about the complex security that they want?

That might still be there, certainly, but that would be active mostly for operations (stuff that happen on a single entity), not on things that happen over all entities. It is drastically easier to make a single entity security decisions work efficiently than make it work over the whole set inside the database.

I was giving a lecture on architecture recently, and the notion of components came in. The most important bit about that lecture was probably at the very end, when I discussed what it is that I consider to be a component. During that discussion, I introduced Monika, the payment processing component.

Monika has the following Service Level Agreement:

• Payment initiation is done messages.
• Notification about payment completion is handled via a callback REST call.
• The SLA calls for 90% of all successful payments to be processed in 2 business days.

So far, it doesn’t sound really complicated, right? And there isn’t even a hint of how Monika works in the SLA or the contracts.

This is Monica:

Well, not really, but it makes the point, doesn’t it.

Monica is a component in the system that respond to (SMTP) messages, does some work, and respond by clicking on a link in the email (REST call).

Monica has a really sucky SLA, since she has only 22% uptime over the course of the year, and then there are those two weeks when she has her yearly maintenance period (vacation), etc.

The most important thing about this is that we are able to abstract all of that away and treat this scenario as just another component in the system.

All too often, people hear components and they start thinking about things like this:

A component in a system is usually something much larger than a single class or a set of classes. It is an independent agent in the system that has its own behavior, resources, dedicated team and deployment schedule separate from all other components.

This is part of the RavenDB test suite.

It has a bug.

Can you see it?

One of the things that I learned when working on the Castle project is that errors are important. In fact, they are really important, so it is worth the time to check them very carefully.

In this case, this code would fail but won’t tell me why it is failing.

Changing it to this:

Means that I get the actual error message right there and then, no need to do anything special.

In my previous post, I asked: What are the performance implications of the two options?

Versus:

And the answer is quite simple. The chance to optimize how it works.

In the first example, we have to return an unknown amount of information. In the second example, we know how much data we need to return. That means that we can optimize ourselves based on that.

What do I mean by that?

Look at the method signatures, those requires us to scan a secondary index, and get the results back. From there, we need to get back to the actual data. If we knew what the size of the data that we need to return is, we could fetch just the locations from the index, then optimize our disk access pattern to take advantage of sequential reads.

In the first example, we have to assume that every read is the last read. Callers may request one item, or 25 or 713, so we don’t really have a way to optimize things. The moment that we have the amount that the caller wants, things change.

We can scan the index to get just actual position of the document on disk, and then load the documents from the disk based on the optimal access pattern in terms of disk access. It is a very small change, but it allowed us to make a big optimization.

What are the performance implications of the two options?

Versus:

There is a level of tension between what a developer wants and what the client wants. As a developer, I want to use the coolest technologies, the freshest methodologies, to be so far on the bleeding edge that they have an ER dedicated just for my team.

As a client? I want to be as conservative as possible, for the most part, I don’t care for new technologies unless they have some significant feature that I can’t get anywhere else. I value speed of development over having the devs geek out.

I spoke about before, the tendency of developers to start building castles in the sky in favor of getting things done. I call it Stealing From Your Client, because you are introducing additional things that aren’t necessary, won’t bring additional value or just plain make things hard.

For example, in just about any project that I have seen, putting an IRepository in front of an OR/M or putting an IRepository in front of RavenDB was a mistake. It created no value and actually made things harder to use.

Other pet peeves of mine was raised in a discussion about architecture in a recent lecture that I gave. One of the participants asked me for my opinion about CQRS. But when I asked him for what sort of an application, he answer was somewhere along the lines of: “I want to build my next application using CQRS and I like to hear your opinion.”

This sort of thinking drives me bananas. (Editor note: That sounds like a very painful thing, and I am not sure why I would do that, but it appears that this is an English expression that fit this location in the post).

Dictating the architecture of an application, before you even have an application? Hell, making any decisions about the application (including what technologies to use) before you actually have a good idea about what is going on is a Big No! No!

Here is the most important piece of this post. For the most part, all those cool technologies that you hear about? They aren’t relevant for your scenario. And even if you are convinced that they are a perfect fit, there is a cost associated with them, which you have to consider before attempting them. Especially if this is the first time that you are using something.

The really annoying part? Most people who come up with these exciting new technologies spent an inordinate amount of their time saying not how to build apps using them, but when you shouldn’t. Evans’ DDD book quite explicitly state that DDD shouldn’t be your default architecture, that it has a cost and should be carefully considered. The industry as a whole (me included) just ignored him and started to write DDD applications.

Greg Young just lectured in Oredev about why you shouldn’t use CQRS. For much the same reason.

At the end, we are there to provide value to the customer. Yes, it is cool to work on the newest thing on the block, but that is why you have hobby projects.

With regards to my recommendation to avoid the repository, Stan asks:

You returned to explain why repository pattern is evil. Just interesting to know what are you doing when you need in your model to access another aggregate. Do you reference NH from your model? I prefer to leave my model POCOed and wrap DB calls by repository pattern. Sleeping good with it.

This is an interesting question. There are several ways to answer that. To start with, assuming that we are using DDD model (which the usage of a repository would imply), you don’t have references between aggregates.

But let us assume that we somehow need that, Stan seems to suggest something like this proposed solution:

public class Person 
{
    public static readonly DateTime ImportantDate;
    public BirthPlace BirthPlace { get; set; } 

    public DateTime BirthDate 
    { 
        get; private set;
    } 

    public void CorrectBirthDate(IRepository<BirthPlace> birthPlaces, DateTime date)
    {
        if (BirthPlace != null && date < ImportantDate && BirthPlace.IsSpecial) 
        { 
            BirthPlace = birthPlaces.GetForDate(date); 
        }
    }
}

Here we have a business rule that states that this is required.

But do we actually need a repository here? What if we just said that whoever calls us need to provide us with a way to get the birth place by date?

public void CorrectBirthDate(
        Func<DateTime, BirthPlace> getBirthPlaceFordate, 
        DateTime date)
{
    if (BirthPlace != null && date < ImportantDate && BirthPlace.IsSpecial) 
    { 
        BirthPlace = getBirthPlaceFordate(date); 
    }
}

This can be done with a simple delegate, no need to introduce a heavy weight abstraction. This is a local solution for a local problem. It keeps the database out from your entities and more importantly, it allows you to actually craft the appropriate response to this at the time of the call.

This is a review of the Northwind Starter Kit project, this review revision 94815 from Dec 18 2011.

A while ago I said:

Seriously?!  22(!) projects to do a sample application using Northwind?

And people took me up to task about it. The criticism was mostly focused on two parts:

• I didn’t get that the project wasn’t about Northwind, but about being a sample app for architectural design patterns.
• I couldn’t actually decide that a project was bad simply by looking at the project structure and some minor code browsing.

I am sad to say that after taking a detailed look at the code, I am even more firmly back at my original conclusion.  I started to do a review of the UI code, but there really is no real need to do so.

The entire project, as I said in the beginning, is supposed to be a sample application for Northwind. Northwind is a CRUD application. Well, not exactly, it is supposed to be an example of an Online Store, which is something much bigger than just Northwind. But it isn’t.

Say what you will, the Northwind Starter Kit is a CRUD application. It does exactly that, and nothing else. It does so in an incredibly complicated fashion, mind, but that is what it does.

Well, it doesn’t do updates, or deletes, or creates. So it is just an R application (I certainly consider the codebase to be R rated, not for impressionable developers).

If you want to have a sample application to show off architectural ideas, make sure that the application can actually, you know, show them. The only thing that NSK does is loading stuff from the database, try as I might, I found no real piece of business logic, no any reason why it is so complicated.

So, to the guys who commented on that, it isn’t a good project. If you like it, I am happy for you, there are also people who loves this guy:

Personally, I would call pest control.

With regards to my quests against repositories, Matt asks:

…if my aggregate root query should exclude entities that have, for example, and IsActive = false flag, I also don't want to repeatedly exclude the IsActive = false entities. Using the repository pattern I can expose my Get method where internally it ALWAYS does this.

The problem with this question is that it make a false assumption, then go ahead and follow on that false assumption. The false assumption here is that the only way to handle the IsActive = false in by directly querying that. But that is wrong.

With NHibernate, you can define that with a where condition, or as a filter. With RavenDB, you can define that inside a query listener. You can absolutely set those things up as part of your infrastructure, and you won’t need to create any abstractions for that.

This is a review of the Northwind Starter Kit project, this review revision 94815 from Dec 18 2011.

Okay, enough about the data access parts. Let us see take a look at a few of the other things that are going on in the application. In particular, this is supposed to be an application with…

Domain logic is implemented by means of a Domain Model, onto a layer of services adds application logic. The model is persisted by a DAL designed around the principles of the "Repository" patterns, which has been implemented in a LINQ-friendly way.

Let us try to figure this out one at a time, okay?

The only method in the domain model that have even a hint of domain logic is the CalculateTotalIncome method. Yes, you got it right, that is a method, as in singular. And that method should be replaced with a query, it has no business being on the domain model.

So let us move to the services, okay? Here are the service definitions in the entire project:

Look at the methods carefully. Do you see any action at all? You don’t, the entire thing is just about queries.

And queries should be simple, not abstracted away and made very hard to figure out.

The rule of the thumb is that you try hard to not abstract queries, it is operations that you try to abstract. Operations is where you usually actually find the business logic.

This is a review of the Northwind Starter Kit project, this review revision 94815 from Dec 18 2011.

One of the things that I repeatedly call out is the forwarding type of architecture, a simple operation that is hidden away by a large number of abstractions that serves no real purpose.

Instead of a controller, let us look at a web service, just to make things slightly different. We have the following:

Okay, let us dig deeper:

I really like the fact that this repository actually have have FindById method, which this service promptly ignores in favor of using the IQueryable<Customer> implementation. If you want to know how that is implemented, just look (using the EF Code First repository implementations, the others are fairly similar):

 

All in all, the entire thing only serves to make things harder to understand and maintain.

Does anyone really think that this abstraction adds anything? What is the point?!

With regards to my quests against repositories, Matt asks:

For example, you dismiss the repository pattern, but what are the alternatives? For example, in an ASP.NET web application you have controllers. I do NOT want to see this code in my controllers:

var sessionFactory = CreateSessionFactory();

using (var session = sessionFactory.OpenSession()) { using (var transaction = session.BeginTransaction()) { // do a large amount of work

// save entities
session.SaveOrUpdate(myEntity);

transaction.Commit();

} }

That is ugly, repetitive code. I want in my service methods to Get, update, save, and not have to worry about the above.

This is a straw dummy. Set up the alternative as nasty and unattractive as possible, then call out the thing you have just set up as nasty and unattractive. It is a good tactic, except that this isn’t the alternative at all.

If you go with the route that Matt suggested, you are going to get yourself into problems. Serious ones. But that isn’t what I recommend. I talked about this scenario specifically in this post. This is how you are supposed to set things up. In a way that doesn’t get in the way of the application. Everything is wired in the infrastructure, and we can just rely on that to be there. And in your controller, you have a Session property that get the current property, and that is it.

For bonus points, you can move your transaction handling there as well, so you don’t need to handle that either. It makes the code so much easier to work with, because you don’t care about all those external concerns, they are handled elsewhere.

This is a review of the Northwind Starter Kit project, this review revision 94815 from Dec 18 2011.

The database is usually a pretty important piece in your application, and it likes to remind you that it should be respected. If you don’t take care of that, it will make sure that there will be a lot of pain in your future. Case in point, let us look at this method:

It looks nice, it is certainly something that looks like a business service. So let us dig down and see how it works.

It seems like a nice thing, the code is clear, and beside the bug where you get 100% discount if you buy enough and the dissonance between the comment and the code, fairly clear. And it seems that we have service logic and entity logic, which is always nice.

Except that this piece of code issues the following queries (let us assume a customer with 50 orders).

1 Query to load the customer, line 34 in this code. And now let us look at line 35… what is actually going on here:

Okay, so we have an additional query for loading the customer’s orders. Let us dig deeper.

And for each order, we have another query for loading all of that order’s items. Does it gets worse?

Phew! I was worried here for a second.

But it turns out that we only have a Select N+2 here, where N is the number of orders that a customer has.

What do you want, calculating the discount for the order is complicated, it is supposed to take a lot of time. Of course, the entire thing can be expressed in SQL as:

SELECT 
  SUM((UnitPrice * Quantity) * (1 - Discount) Income
FROM OrderItems o
WHERE o.OrderID in (
  SELECT Id FROM Orders
  WHERE CustomerId = @CustomerId
)

But go ahead and try putting that optimization in. The architecture for the application will actively fight you on that.

With regards to my recommendation to not use repositories, Remyvd asks:

… if you have several kind of data sources in different technologies, then it would be nice if you have one kind of interface. Also when an object (like Customer) is combined from data out of different data sources, the repository is for me a good place to initialize the object and return it. How would you solve this cases?

My answer is: System.ArgumentException: Question assumes invalid state.

More fully, this is one of those times where, in order to actually answer the question, we have to correct the question. Why do I say that?

Well, the question makes the assumption that actually combining the customer entity out of different data stores is desirable. Having made that assumption, it proceed to see what is the best way to do that. I am not going to recommend a way to do that, because the underlying assumption is wrong.

If your Customer information is stored in multiple data stores, you have to ask yourself, is it actually the same thing in all places? For example, we may have Customer entity in our main database, Customer Billing History in the billing database, Customer credit report accessible over a web service, etc. Note what happens when we start actually drilling down into the entity design. It suddenly becomes clear that that information is in different data stores for a reason.

Those aren’t the druids you are looking for might be a good quote here. The fact that the information is split usually means that there is a reason for that. The information is handled differently, usually by different teams and applications, it deals with different aspects of the entity, etc.

Trying to abstract that away behind a repository layer loses that very important distinction. It also forces us to do a lot of additional work, because we have to load the customer entity from all of the different data stores every time we need it. Even if most of the data that we need is not relevant for the operation at hand.

If would be much easier, simpler and maintainable to actually expose the idea of the multiple data stores to the application at large. You don’t end up with a leaky abstraction and it is easy to see when and how you actually need to combine the different data stores, and what the implications of that are for the specific scenarios that requires it.

This is a review of the Northwind Starter Kit project, this review revision 94815 from Dec 18 2011.

In my previous post, I talked about how the CatalogController.Product(id) action was completely ridiculous in the level of abstraction that it used to do its work and promised to show how to do the same work on an actual read model in a much simpler fashion. Here is the code.

There are several things that you might note about this code:

• It is all inline, so it is possible to analyze, optimize and figure out what the hell is going on easily.
• It doesn’t got to the database to load data that it already has.
• The code actually does something meaningful.
• It only do one thing, and it does this elegantly.

This is actually using a read model. By, you know, reading from it, instead of abstracting it.

But there is a problem here, I hear you shout (already reaching for the pitchfork, at least let me finish this post).

Previously, we have hidden the logic of discontinued products and available products behind the following abstraction:

Now we are embedding this inside the query itself, what happens if this logic changes? We would now need to go everywhere we used this logic and update it.

Well, yes. Except that there are two mitigating factors.

• This nice abstraction is never used elsewhere.
• It is easy to create our own abstraction.

Here is an example on how to do this without adding additional layers of abstractions.

Which means that our action method now looks like this:

Simple, easy, performant, maintainable.

And it doesn’t make my head hurt or cause me to stay up until 4 AM feeling like an XKCD item.

This is a review of the Northwind Starter Kit project, this review revision 94815 from Dec 18 2011.

In my last few posts, I have gone over the data access strategy used in NSK. I haven’t been impressed. In fact, I am somewhere between horrified, shocked and amused in the same way you feel when you see a clown slipping on a banana peel.  Why do I say that? Let us trace a single call from the front end all the way to the back.

The case in point, CatalogController.Product(id) action. This is something that should just display the product on the screen, so it should be fairly simple, right? Here is how it works when drawn as UML:

To simplify things, I decided to skip any method calls on the same objects (there are more than a few).

Let me show you how this looks like in actual code:

Digging deeper, we get:

We will deal with the first method call to CatalogServices now, which looks like:

I’ll skip going deeper, because this is just a layer of needless abstraction on top of Entity Framework and that is quite enough already.

Now let us deal with the second call to CatalogServices, which is actually more interesting:

Note the marked line? This is generating a query. This is interesting, because we have already loaded the product. There is no way of optimizing that, of course, because the architecture doesn’t let you.

Now, you need all of this just to show a single product on the screen. I mean, seriously.

You might have noticed some references to things like Read Model in the code. Which I find highly ironic. Read Models are about making the read side of things simple, not drowning the code in abstraction on top of abstraction on top of abstraction.

In my next post, I’ll show a better way to handle this scenario. A way that is actually simpler and make use an of actual read model and not infinite levels of indirection.

This is a review of the Northwind Starter Kit project, this review revision 94815 from Dec 18 2011.

In my previous posts, I focused mostly on the needlessness of the repositories implementation and why you want to avoid that (especially implementing it multiple times). In this post, I want to talk about other problems regarding the data access. In this case, the sudden urge to wash my hands that occurred when I saw this:

I mean, you are already using an OR/M. I don’t like the Repository implementation, but dropping down to SQL (and unparapetrized one at that) seems uncalled for.

By the way, the most logical reason for this to be done is to avoid mapping the Picture column to the category, since the OR/M in use here (EF) doesn’t support the notion of lazy properties.

Again, this is a problem when you are trying to use multiple OR/Ms, and that is neither required nor really useful.

Okay, enough about the low level data access details. On my next post I’ll deal with how those repositories are actually being used.

This is a review of the Northwind Starter Kit project, this review revision 94815 from Dec 18 2011.

Update: Andrea, the project’s leader has posted a reply to this series of posts.

Yes, this is another repositories are evil if you are using an OR/M post.

That is probably going to cause some reaction, so I am going to back this up with code from this NSK project. Let us talk about repositories, in particular. Let us see what we have here:

Okaay…

Now here are a few problems that I have with this:

• There is no value gained by introducing this abstraction. You aren’t adding any capability what so ever.
• In fact, since all OR/Ms provide an abstraction that isn’t dependent on type, creating IRepository<T> and things like ICustomerRepository is just making things more complicated.
• There are going to be changes in behavior between different repositories implementations that will break your code.

Let us see what we actually have as a result. This is the Entity Framework POCO implementation:

You can probably guess how the rest of it is actually implemented. Yes, we have a LOT of code that is dedicated solely for this sort of forwarding operations.

And then we have the actual implementation of the delete:

Just to remind you, here is the NHibernate implementation of the same function:

Leaving aside the atrocious error handling code, the EF POCO version will do an immediate delete. The NHibernate version will wait for the transaction to be committed.

And don’t worry, I do remember the error handling. This is simply wrong.

And then we have implementations such as this:

This is for the Entity Framework Code First implementation. There is a message here that is coming to me loud and clear. This code wants to be deleted. It is neglected and abused and doesn’t serve any purpose in life except gobble up pieces of valuable disk space that could be filled with the much more valuable result of  reading from/dev/random.

Steve Py asks an interesting question in one of the comments to my On Infinite Scalability post:

Can you elaborate more on: "Note, those changes are not changes to the code, they are architectural and system changes. Where before you had a single database, now you have many. Where before you could use ACID, now you have to use BASE. You need to push a lot more tasks to the background, the user interaction changes, etc."

When you talk about jumping from 1 server to multiple servers, ACID to BASE, and how user interaction changes, how do you quantify that this is done without code changes?

The answer to that is that there is a mistaken assumption here. Changing the architecture is going to change the code. But usually that is rarely relevant, because changing the architecture is a big change. If you are moving from a single DB to multiple database, for example, there are going to be code changes, but that isn’t what you worry about. The major change is the architecture differences (how do you split the data, how do you do reporting, can some of the dbs be down, etc).

Moving from ACID to BASE is an even greater change. The code might change a little or change drastically, but that isn’t where a lot of the effort is. Just defining the new system behavior on those scenarios is going to be much more complex. For example, taking something as simple as “user names are unique” would move from being a unique constraint in the database to something that needs to be able to handle those sort of things in a reasonable fashion.

Depending on your original architecture, it might be anything from replacing a single service implementation to re-writing significant parts of the code.

Udi Dahan posted about the myth of infinite scalability. It is a good post, and I recommend reading it. I have my own 2 cents to add, though.

YAGNI!

When building a system, I am always assuming an order of magnitude increase in the work the system has to do (usually, we are talking about # of requests / time period).

In other words, if we have 50,000 users, and we have roughly 5,000 active users during work hours, we typically have 100,000 requests per hour. My job is to make sure that we can manage to get up to 1,000,000 requests per hour. (Just to give you the numbers, we are talking about moving from ~30 requests per second to ~300 requests per second).

Why do we need that buffer?

There are several reasons to want to do that. First, we assume that the system is going to be in production for a relatively long time, so the # of user or their activity is going to grow. Second, in most systems, we are usually talking about some % of the users being active, but there are usually times when you have a significantly more users being active. For example, at end of tax year, an accounting system can be expected to see a greater utilization, as well as at every end of the month.

And within that buffer, we don’t need to change the system architecture, we just need to add more resources to the system.

And why a full order of magnitude?

Put simply, because it is relatively easy to get there. Let us assume end of year again, and now we have 15,000 active users (vs. the “normal” 5,000). But the amount of work we do isn’t directly related to the number of users. It is related to how active they are and what operations they are doing. In such a scenario, it is more likely that the users will be more active and stress the system more.

Finally, there is another aspect. Usually, moving a single order of magnitude up is no problem. In fact, I feel comfortable running on a single server (maybe replicate for HA) from 1 request per second to 50 or so. That is 3,600 request per hour to 180,000 requests per hour. But beyond that, we are going to start talking about multiple servers. We can handle a millions requests per hour on a web farm without much problems, but moving to the next level is likely to require more changes, and when you get beyond that, you require more changes still.

Note, those changes are not changes to the code, they are architectural and system changes. Where before you had a single database, now you have many. Where before you could use ACID, now you have to use BASE. You need to push a lot more tasks to the background, the user interaction changes, etc.

Of course, you can probably start your application design with a 10,000,000 requests per hour. That is big enough that you can move to a hundred million requests per hour easily enough. But, and this is important, is this something that is valuable? Is this really something that you can justify to the guys writing the checks.

Scalability is a cost function, as Udi has mentioned. And you need to be aware that this cost incurs during development and during production. If your expected usage can be handled (including the buffer) with a simpler architecture, go for it. If you grow beyond an order of magnitude, there are three things that will happen:

1. You have the money to deploy a new system for the new load.
2. You now have a better idea on the actual usage, rather than just guessing about the future.
3. You are going to change how the system work anyway, from the UI to the internal works.

The last one is important. I am guessing that many people would say “but we can do it right the first time”. And the problem is that you really can’t. You don’t know what users will do, what they will like, etc. Whatever you are guessing now, you are bound to be wrong. I have had that happen to me on multiple systems.

More than that, remember time to market is a big feature as well.

When I am thinking about modern art, I am thinking about an experience that I had that was remarkably similar to this one:

I got a lot of comments regarding my review of the Northwind Starter Kit project.

Here is the deal, if you want to demonstrate complex ways to solve a problem, you had better make sure that you are actually solving a problem that requires a complex solution. If you are demonstrating how to solve a simple problem in a complex way, you are basically doing disservice to the reader.

When I wanted to write a sample app to demonstrate something, I either chose to demonstrate the actual technology (writing a ToDo app) or I spent dozens of posts establishing the context (yes, that is Macto, I’ll get back to it).

But since so many people seems to have been offended by my slight of dismissing the project based on what seemed like just the number of projects, I’ll do a full review series on that. My point was to make it clear that creating complex solutions for simple problems is wrong, especially if you are trying to demonstrate a real workable system. Without the proper context, all of this stuff is just cargo cult.

At several customer visits recently, I encountered a common problem. Their production error handling strategy. In one customer, they capture all of the production errors, push them to System Center Operations Manager (SCOM) and then send those errors to the development team over email. On another customer, they didn’t really have a production error handling strategy.

The interesting thing about that is that in both cases, production errors were handled in exactly the same way. They were ignored until a user called and complained about that.

Huh?!

What do you mean, ignored? The first client obviously did the right thing, they had capture and monitored the production errors, notifying the development team on each and every one of them.

Well, it is actually very simple, at the first client, I asked everyone to raise their hands if they receive the production errors emails. About half the room raised their hands. Then I asked how many of them set up a rule to move those emails from their inbox.

Every single one of them had done that.

The major problem is that errors happen all the time. In the vast majority of cases, you don’t really care, and it will fix itself automatically. For example, a error such as Transaction Deadlock Exception might happen a few times a day. There really isn’t much you can do about those errors (well, re-architecting the app might do that, but that is out of scope for this post). Another might be a call to an external service that occasionally fails and already have a retry strategy in place.

Do you get the problem?

Getting notified about every single production error has immunized the team from them. Now going over the productions errors is just a chore, and a fairly unpleasant one.

That is a major difference between Errors and Alerts. An error is just an exception or a problem that happened in production. Usually, those aren’t really important. It will sort itself out. An ETL process that runs once an hour can fail a few times, and as long as it’ll complete within a reasonable time frame, you don’t care.

Did you notice how often that statement is repeated. You don’t care.

When do you care?

• When the ETL process has been unable to complete for three consecutive times.
• When the external service that you call has been unresponsive for over 8 hours.
• When a specific error is happening over 50 times an hour.
• When an unknown error showed up in the logs more than twice in the last hour.

Each of those cases requires a human intervention in needed. And in most cases, those are going to be rare.

Errors are common place, they happen all the time and no one really care. Alerts is what you wake up at 2 AM for.