I speak quite often about DSL and how to build them, but so far I did not do was explain why you need a DSL at all. After all, since you are reading this blog, you already know how to program. Can’t we just use “normal” programming languages to do the same job? We can even do with a dash of fluent interfaces and Domain Driven Design to make the code easier to read.

We need to inspect the different needs that lead the drive toward a DSL.

A Technical DSL

A technical DSL is supposed to be consumed by someone that understands development. It is meant to express matters in a more concise form, but it is still very much a programming language at heart. The main difference is that it is a language focused on solving the specific problem at hand. As such, it has all the benefits (and drawbacks) of single purpose languages. Examples for technical DSL include Rake, Binsor, Rhino ETL, etc.

The driving force around that is that you want to have richer expressiveness in specifying what you want to happen. Technical DSL are usually easier to write, because your target audience already understands programming.

In fact, the use of programming features can make a DSL very sweet indeed. We have already seen a Rake sample, so let us see a Binsor sample:

for type in Assembly.Load(“MyApplication.Web”).GetTypes():
     continue unless type.IsAssignableFrom(Controller)
     component type.FullName, type

So we take three lines to register all the controllers in the application. That is quite a bit of expressiveness. Of course, it assumes that you are familiar with the API and its usage, which is not always true, which leads nicely to business focused DSL

A Business DSL

A business DSL is focused on being (at least) readable to a businessperson with no background in programming.

This type of DSL is mainly expressive in the terms of the domain, and it has a lot less emphasis on the programming features that may still exists. It is also tend to be much more declarative than technical DSL in general and a lot more emphasis is placed on the nature of the DSL so those would not really be necessary.

I can’t really think of a good example of a business DSL in the open. A good example of DSL that I have run into include a cellular company that had to have some way to handle all the myriad of different contracts and benefits that it had. It also needed to handle this with a fairly rapid time to market, since the company needed to respond quickly to market conditions.

The end result was a DSL in which you could specify the different conditions and their results. For instance, to specify that you get 300 minutes free if you speak over 300 minutes a month, you would write something similar to this:

if CallMinutesForCurrentMonth > 300 and HasBenefit “300 Minutes Free!!!”:
          AddCallMinutes -300, "300 Minutes Free!!!"

It was fairly simple to define a small language that could describe most of the types of benefits that the company wanted to express. The rest was a matter of naming conventions and dropping files in a specified folder, to be picked up and processed at regular intervals. The structure that surrounds a DSL is a subject that deserves quite a bit of attention on its own.

A businessperson may not always be able to write actions using a business DSL (more on that later), but they should be able to read and understand it. After all, it is their business and their domain that you are trying to describe.

Now, why shouldn’t a businessperson be able to write actions using a business DSL?

The main reason, as I see it, is of error handling. No, I don’t mean in the actual running of the DSL action, I mean when writing it.

A DSL is supposed to be read like a language, but it is still a programming language, and those have little tolerance for such thing as bad casing for keywords, for instance. Certain types of users will simply be unable to go over the first hurdle in the road they face, because of this.

It is important to know your audience, and it is even more important not to be contemptuous toward that mythical businessperson. You may not think that this person can understand programming, only to discover that there are quite a bit of automation going on in their life already, powered by VBScript and excel macros.

If you can leverage this knowledge, you have a very powerful combination in your hand, because you can provide that businessperson the tools, and he can provide the knowledge and the required perspective.

Automation DSL

I am not quite sure about this classification, but it certainly has its place. Another name for this may be the IT DSL. This type of a DSL it often used to expose the internal of an application to the outside world.

Modern games are usually engines that are being configured using some sort of a DSL. In fact, I fondly remember building levels in Neverwinters Nights.

More serious usage for this can certainly be found, such as a DSL that lets you go into the internals of an application and mange it. Think about the ability to run a script that will take re-route all traffic from a server, wait for all current work to complete, and then take the server down, update it and bring it up again.

Right now, it is possible to do this with shell scripts of various kinds, but most enterprise application can certainly have more visibility into them than already exist, and a DSL that will allow me to inspect and modify the internal state would be very welcome.

I can certainly think of a few administrators who would be grateful to have more power in their hands.

To conclude, can you think of other types of DSL in use?

Andrey Shchekin made a good point when he commented on my post about graphical DSL:

I do not use DSLs that are purely graphical. But talking about documentation, I think it is very useful to have a graphical _representation_ for some DSLs that is synchronized with (or generated from) the DSL itself.
For example, any workflows or dataflows (think Rhino ETL) are much easier to see at a glance on visual surface. Editing them on visual surface is also an option, but not a requirement.

I certainly am a believer that a DSL should be mostly declarative, and indeed, taking Rhino ETL as an example, it is something that can have a visual representation.

The only remaining question is how?

Long ago there was Lithium and Netron projects, which were good for building the kind of UI that a DSL is composed of. Those have gone commercial now, and while I can probably find the originals somewhere, that is not a very promising concept. There is always the Visual Studio DSL toolkit, but I have a strong suspicion that I would not like it (COM involved), and anyway, I want to do it externally to Visual Studio.

Anyone familiar with useful libraries that can make it happen? Ideally, it would allow me to generate UI similar to the class designer. It should also be simple, but I am not sure how to really quantify that.

Another form of DSL is the graphical domain specific language. 

What do I mean by that? I am talking a DSL that is not textual, but rather uses shapes and lines in order to express intent.

A good example would be UML. That is a DSL for describing software systems. In fact, quite a lot of money and effort has been devoted to making UML the One True Model, from which you can generate the rest of the application.

There has been also a lot of effort invested in making it possible to write your own graphical DSL. Microsoft has the Visual Studio DSL Tools, which is a framework that allows you to build tools similar to the class designer and generate code off of them.

Existing examples of Graphical DSL that comes to mind are:

· UML

· BizTalk

· Sql Server Integration Services

· Workflow Foundation

I don’t like graphical DSL. To be rather more exact, I simply love graphical DSL for documentation, but I find that they make a rather poor job when it comes to actual development.

I have some experience with all of the above, and I can say confidently that they all share common problems, and that those are inherit to the graphical DSL model.

For a start, any problem of any real complexity would be very hard to build using a graphical DSL. Just arranging all the shapes in a way that you can browse them easily takes significant bits of time, and then you get to the issue of actually understanding what this thing does.

The whole point of a graphical DSL is to hide information that you don’t want to see, to let you see “big picture”. All of which means that you can’t really see the whole at the same time, and this leads to a lot of time spent jumping between various elements on the DSL surface, trying to gather all the required data.

And then there are UI issues that are important. How do I do a search & replace operation in a graphical DSL? How can I just look for something? And, of course, there is a reason why mouse driven development is not a good idea, if only for your wrist.

Beyond that, there are serious issues with how do you make a reasonable diff between two versions. No, reading XML files diff (assuming they are even diffable) don’t work, you need some way to express diffs in a graphical way, and so far I haven’t seen any good way to do that.

From an implementation perspective, there are other issues that you need to consider. In a graphical DSL, the need to express things visually is pretty important, so we need to have some conventions about what shapes and connections we have. The actual what and how will depend on the actual language that you are trying to implement.

If you haven’t guessed so far, I am not a fan of graphical DSL. Not for programming, at least, it is a valuable tool for documentation and design, but quite a few people are pushing it too hard.

Steve is talking about tasers and taser related deaths. 

I remember when we got a few tasers. The rules for using them were pretty strict, they were considered as firearm for all intents and purposes. Using them as a method to "subdue" people once they are handcuffed would fall under the category of torture. I can just imagine what I, personally, would do to anyone who tried that, and I would be the only first step in the chain.

Then, of course, you have the nice legal situation where we literally had to get a doctor's permit to use it. The open fire conditions where the same, but if the inmate had a heart problem, we were to shoot using old fashion bullets rather than use a taser. Old fashion firearms didn't need a doctor's permit to shoot.

This is a screwed up world. Then again, we never gotten into a situation where we actually had to shoot someone, for which I am grateful. I abhor violence. It begets too much paperwork.

If deployment means a week with little sleep and beepers going off just as you nod to sleep, then you are doing it wrong. I am saying that as someone whose standard deployment procedure has two steps, "svn up" and "build.cmd". Actually, I lied, I have a single step, deploy.cmd, which runs them both for me.

Now, you can't always get this procedure to work, this is especially true if you need to deploy several different parts of the same application at the same time. The classic example is making a change that requires modifying the database as well. More interesting examples consist of several layers of the same application, web servers, application servers, dedicated tasks servers and the database servers, all of whom need to be touched in order to deploy the new version of the application.

One of my customers takes the simple approach of shutting down everything and systematically upgrading all the systems when the application itself is offline. That is probably the simplest approach, you don't have to worry about concurrent state and multi versioning, but it is also the one that is harder to the business to deal with.

In fact, service level agreements may very well limit or prevent doing that. Here is a simple table of the common SLA and their meaning, in terms of system maintenance:

The last number is the one we should pay attention to, it is the magic 5 nines that SLA often revolves around. Now, SLA often talks about unplanned downtime, and in many systems, you can safely take it down outside of business hours and tinker as much as you like with it. Those are not the kind of systems that I want to talk about today. The system that you can take down safely are fairly easy to upgrade in a safe manner, as was previously mentioned.

The systems that are more interesting are the ones that you want to be able to update on the fly, without a painfully long deployment cycle. This is directly related to the ability to meet required SLA, but it also has another critically important factor. Developer productivity. If making a change in the application is something that requires stopping the application, updating files, starting it up again and then being hit with the initial, it is painful, annoying and will lead developers to jump through all sorts of really nasty hops along the way.

So easy deployment is critical both for the developer and for production, but the needs of each are vastly different. For developers, having it work "most of the time" is fine, having it consume more memory is also fine, because the application is not going to run for days on end. For production, it must work all the time, and you must watch your resource consumption very carefully.

A typical hot deployment for production will usually look like this:

This assumes a typical production scenario in which you have duplicates of everything, for fail over and performance.

This is just an extension of the previous cold deployment, since we are taking servers down, just not the application as a whole.

This assumes that we actually can take vertical parts of the application down, which is not always the case, of course.

But, basically, we start with both servers operational, move all traffic to the second server, and then perform a cold deploy to the first set of servers.

Then we move all traffic to the new server and perform cold deployment on the second server.

In the end, this usually means that you can perform the entire thing upgrade without loss of service (although you may suffer loss of capacity.

As I said, this is probably the ideal scenario for perform hot deployments, but it makes quite a few assumptions that are not always valid. The main one is that the application is structured vertically in such a way that we can close and open parts of it and there are no critical points along the way.

I can think of one common scenario where this is not the case, several web servers sitting on top of the same database server, for instance. So, while this is a good approach, we probably want to consider others as well.

Another option is to design the system with multiply concurrent versions from the get go. This gives us the following diagram:

We have several versions running at the same time, and we can migrate users slowly from one version to the next. This also gives us a nice rollback strategy if we have a botched version.

The problem with this approach is that it is a complicated procedure. This is especially true if the application is composed of several independent services that needs to be upgraded separately. Again, database and web servers are common example, but even web services that needs to be upgraded requires some interesting mechanics when you have enough of them.

I am going to talk about multi versioned databases in the next post in the series, so let us just take as a given that this is possible, and consider the implications.

If I can run multiply versions at the same time, this means that I can probably propagate a change throughout a cluster of servers without having to worry about the change over semantics. (The time between the first server getting the change and the last one getting it).

Before we go on with production deployments, I want to talk a bit about developer deployments. What do I mean by that?

Well, deployment is probably the wrong term here, but this is the way most of the enterprise platforms make you go about it.  And that is a problem. Deploying your changes just in order to run them is a problematic issue. It is problematic because it interferes with make-a-change/run approach, which is critically important for developer productivity.

As mentioned above, any multi step process is going to be painful as hell over any significant development. I often have to go through several dozens of change/run before I can say that I have finished with a particular feature.

There is a strong incentive to make it as easy as possible, and as it turns out, we actually have the tools in place to do it very easily indeed. I have produced a screen cast that talks about hot deployment of compiled code, and applying the same principals to code in its textual form is demonstrated here.

Here is the directory structure that I envisioned for the platform. As you can guess, this is backed by source control, but more importantly, it is alive (queue mad laugher here).

But what do I mean by that? I mean that any change whatsoever that is done to this directories will be automatically picked up and incorporated by the system as soon as the change is detected.

This supports very nicely the idea of change & run mentality. But what actually goes into these folders?

The parts that are important to us are the entities, controllers and views. For discussing those, we will take the Account entity as an example.

For the system, an Account entity is composed of the following files:

There is a reason why I choose to use boo as the primary language for extending the platform. Not just because of personal bias (which exists), but because it makes it so much easier to deal with quite a few issues.

One of them is the ability to define my own syntax, so the content of the Account.boo file would be similar to this:

entity Account:
    AccountId as int
    Name as string

The ability to define entity as a keyword means that I don't need to do anything more to define any persistence concerns, even though I intend to use this as an Active Record class, it is all handled by the framework.

I do intend to allow extension using compiled code, that is why the binaries folder is there for, and you can certainly define C# classes, but the main idea here is to allow easy editing of the whole thing, which means that compilation is not necessarily a good thing.

So, after this explanation, let us go back a bit and talk about what deployment means in this scenario? Well, the first thing that it means is that once a change is detected, you want to recompile the file and keep on moving without breaking your stride. Brail itself is a good example of it. Brail templates are instantly updated if changed, but they are usually compiled (thus supposedly faster). From the developer perspective, there isn't any difference between the two. It works very well in practice, and the productivity boost means that it is encourage the small steps approach. All in all, I am quite happy with it.

I am going to leave the technical details aside for now, let us just say that it is equally easy to do in both source and binary form, and you can either see the webcast or check the post about each one.

There are a few things that we should be worried about, however, mainly, recompiling files all over the place will cause an assembly leak, which can have an adverse affect on memory consumption. Here we get to the some interesting design decisions. Batching compilation will help greatly here, so we can grab all the controllers and compile them into a single assembly, then recompile the changes into separated assemblies. This is the solution used by Brail, for instance.

This seems like it can cause problems, and in theory, it will, but in practice, there are several mitigating factors:

• During development, we can expect the application lifetime to be short enough that assembly leakage is rarely an issue, if it is, there is a small cost to restarting the application.
• On production, we will rarely expect to have a lot of churn, so we can handle the extra memory requirement (in the order of a single megabyte or so, in most cases).

More advance scenarios calls for either AppDomain restart (the way ASP.Net does it) or a separate AppDomain that can be safely loaded. Personally, I think that this would make the situation much harder, and I would like to avoid it if possible. The simplest solution works, in my experience.

What this all means is that a developer can go and make a change to the Account controller, hit the page and immediately get the changes made. Deployment now means that we commit to the development branch, merge to the production branch, and then we request the production system to update itself from the source. A form of CI process is a valid idea in this scenario, and will probably be the default approach to updating changes in a single system scenario. We have to have a way to disable that, because we may want to upgrade only some of the servers at a time.

This leaves us with the following issues:

Debugging - How can we debug scripts? Especially dynamically changing scripts? Despite what some of the Ruby guys say, I want to have the best debugger that I can available. A platform with no or poor debugging support has a distinct disadvantage.

As it turns out, the problem isn't that big. The Visual Studio debugger is a really smart one, and it is capable of handle this in most cases with aplomb. And since Boo code is compiled to IL, VS has few issues with debugging it. For the more complex scenarios, you can use C# and just direct the build path to the binaries folder.

In any case, debugging would usually involve "Attach to process" rather than "Start with debugger", but this is something that I can live with (Attach is faster anyway).

Database modifications - let us just hand wave the way we are going to handle that for a minute. We will just assume we can do something like UpdateSchema() and it will happen, we still need to think about the implications of that.

Now we need to think about how we are going to handle that when we update an entity. Do we want this update to the schema to be automatic, or do we want it to happen as a result of user input? Furthermore, changing the entity basically invalidate the ability to call the database, so how do we handle that?

Do we disable the usage of this entity until the DB is updated? Do we just let the user to run into errors? What do we do for production, for that matter? I definitely not going to let my system run with DDL permission for production, so that is another problem.

I can give arguments for each of those, but a decision still has to be reached. Right now I think that the following set of policies would serve well enough:

• For production, all database changes must be handled after an explicit user action. Probably by going to a page and entering the credentials of a user that can execute DDL statements on the database.
• For development, we will support the same manual approach, but we will also have a policy to auto update the database on entity change.

We are still in somewhat muddy water with regards to deploying to production with regards to changes that affects the entire system, to wit, breaking database changes. I am going to discuss that in the next installment, this one got just a tad out of hand, I am afraid.

Here is an interesting question, at what point you drive away so far from the original language that you lose the benefits of an internal DSL?

The following examples are several ways to represent the same thing, going from the extreme DSL to the no DSL to C#. I think that they all have value, and neither cross that line, but I believe that they will help make the point. The example here is a a simple MonoRail Controller:

#1:

controller Field:
    action Tag:
           @defaultTags = [ "green", "trees", "growth" ]

#2:

controller Field:
    def Tag():
         @defaultTags = [ "green", "trees", "growth" ]

#3:

controller Field:
    def Tag():
          PropertyBag["defaultTags"] = [ "green", "trees", "growth" ]

#4:

class FieldController(Controller):
   def Tag():
          PropertyBag["defaultTags"] = [ "green", "trees", "growth" ]

#5:

public class FieldController : Controller
{
	public void Tag()
	{
		PropertyBag["defaultTags"] = new string[] { "green", "trees", "growth" };
	}
}

Any comments?

I have been doing some work with MonoRail recently, and I noticed that I am structuring the UI in a completely different way than the way I would using WebForms.

When I used WebForms, I at first tried to make significant use of the builtin controls and components, thinking that they would make my work easier. But it turned out that they added complexity instead of removing it. Then I tried to use WebForms in as pure HTML generation capacity as possible. I made very big use of repeaters and such controls, some of which I built myself.

When I just started to use MonoRail, I had much the same style, but now I am finding out that I am doing things differently. The more recent solutions relies a lot more about generating JS for data, and then generating the UI using JS. The somewhat primitive approach outlined below represent a spike for a new feature, where we simply wanted to see how we can get it started, but I think that this makes the point fairly obvious.

<script type="text/javascript">
	var orders = [];
	var lines = [];
<% 
	for order in orders: 
%>
	orders.push({
			id: "!{order.Id}",
			name: "!{order.Name}",
		});
<%
		for orderLine in order.OrderLines: 
%>
	lines.push({
		id: "!{orderLine.Id}",
		orderId: "!{order.Id}",
		orderName: "!{order.name}",
		cost:	"!{orderLine.Cost}",
		account: "!{account.name}" 
	});
<% 
	end 
end
%>

Event.observe(window, 'load', function(){
	for(var i = 0; i< lines.length; i++)
	{
		var line = lines[i];
		var td = $('orderlinessTR').down('td');
		var div = document.createElement("div");
		div.className = "order";
		div.tag = line;
		div.innerHTML = line.orderName;
		td.appendChild(div);
	}
	
	for(var i = 0;i<orders.length; i++)
	{
		$('ordersDDL').options.add(new Option(orders[i].name, orders[i].id));
	}

	Event.observe($('ordersDDL'), 'change', function(){
		var e = $('ordersDDL');
		var id = e.options[e.selectedIndex].value;
		var divs = $$('div.line');
		for(var i = 0; i< divs.length; i++)
		{
			if(divs[i].tag.orderId == id)
				divs[i].style.background = "red";
		}
	});
});

</script>

As you can see, we start by generating JS for the data, and then process it somehow. I considered using JSON to serialize my entities directly to JSON strings, but I am absolutely against tying the javascript to my domain model. An explicit translation step is much more welcome in this case, although the one above it is not a very robust one, I will admit.

In fact, as a direct correlation of this trend, I find myself doing a lot more work that centers around creating javascript controls. Those are rich, incredibly easy to develop, frustrating at times (usually when I am working on explorer) and framework independent. I have completely buy into Prototype/JQuery (I really should post about that) as an underlying framework to work with, but I have yet to start working with the JS control libraries that I hear other peoples raving about. I made some use of scriptacolous, but that is about it.

Am I doing something strange? Or is it just a common trend that I missed?

At first, there was the bit. And the bit shifted left, and the bit shifted right, and there was the byte. The byte grew into a word, and then into double word. And the developer saw the work and it was good. And the evening and the morning were the first day.

On the next day, the developer came back to the work and couldn’t figure out what he was thinking last night, and spent the whole day figuring that out.

After reading my post about enterprise security infrastructure, Comlin asks:

Ayende, What would you say if I give you a real world scenario:
1. Server running Microsoft Windows hosts a web site which includes your security features.
2. Separate server hosts database.
Well, common thing isn't it?
We are pretty sure that the first server(it's on the Internet) would be hacked sooner or later and the hacker will acquire admin privileges. And all we have is your "entity security" inside the BL, who needs it anyway, you will take the connection string and perform all the malicious actions inside the database!
I see two ways to handle this situation: developing an app server to process queries or building security infrastructure inside DB using stored procedures or row-level security. What do you think?

Let us go through several of those iterations in turn, shall we?

First, I disagree with the implicit assumption that it will essentially be broken, and broken with admin privileges. But even then... Well, getting admin access to the box doesn't mean that you get my connection string, by no means. That connection string is DPAPI-ed encrypted and keyed to the user running the application. Even as admin, you have no way to get that out.

Nevertheless, let us assume that a nefarious person did get their hands on a connection string to my database, one that had read and write access. Well, what can it do?

Well, doing something like "UPDATE Sales SET TotalSales = TotalSales * 100 WHERE UserId = @myUserId", and get a hefty commission all of a sudden. But is there really a way to stop it at this point?

It seems to me like at that point, anything more it basically pointless, it is closing the doors with the fox in the coop. Let us take into account the idea of using stored procedures, shall we. What extra security measure do we get from that? The user is already in possession of a connection string with privileges to do things in the database. They can now do things like "while @count  < 100 exec AddSaleTo @myUserId".

Oh, you solve that by using windows authentication all the way to the database, and validate permissions on the logged on user? Well, to do that, the web server must also be marked as trusted for delegation. Which means that if the "hacked" server tells the DB to think that it is Super Mario, it will believe it.

Row level security is bad for the same reasoning as before,  but it has an additional twist. Doing security in the database level utterly exclude the option to cache data. That is bad, really bad.

Then you have the idea of an application server in the middle. Now we have a bit more structure, we may even be able to do some business logic testing on the incoming requests, but, and this is the critical part. The application server has no way to know that the web server was hacked. As far as it is concerned, those requests now being made to it are valid ones, and it will try to honor the thousands AddSaleToUserMessage coming its way.

Now, you could structure you application in such a way that nothing trusted anything else, but that is not a good way to structure an application. To be rather exact, the cost of untrusting the inner circle is very high. Both in terms of the effort involved and as a result of the design, which would be awkward to work with long term.

Now, the diagram here is fairly typical, I would assume, in fact, this is how OS security is built, by having a trusted kernel. The kernel cannot protect itself from itself. That is why rootkits are possible.

And, like rootkits demonstrate, it contains a very problematic flaw, you cannot ensure that all the nodes will remain trustworthy. And when one of them turns red, it is endgame.

A better approach would be to go with the firewall approach, with the external application explicitly not trusted by the internal one, and all communication be inspected for suspicious activities.

You can imagine it a little the diagram to the right. This is another typical architecture design. This time we are going with the application server route, with a firewall (or an application firewall) in the middle.

That point of yellow is an inspection port, in addition to the usual checks a firewall will make, this inspection port will try to analyze the request with a more domain knowledge. It can then trigger alarms or deny access if it suspect that the external application is behaving strangely. Such things that might trigger it are malformed business requests, too much focus on a single item (such as trying to add sales to a salesperson), etc.

Then again, something else that would trigger it might be a data entry clerk that is typing really fast, or a simply sale on the site that means that a lot of people are suddenly buying this one item, etc, etc.

Something that I would definitely be interested to have is an audit trail, so I can revert changes if needed, or at least follow their logic.

In the end, there really isn't a one good way to secure an application, this requires cooperation from developers, IT, networking, etc. If there was, everyone was using it. And while I do believe in defense in depth, I also believe that once the king is taken, the game is over. Starting with the premise that the attacker has gained an admin control over one of your machine is not a start that you want to be in.

It came up in a conversation today. Gmail & OWA are doing much the same thing, using much the same technologies.

Yet it was Gmail who was the Ajax killer application, not OWA.

Why?