My previous approach isn't really suited to anything but the most trivial of tasks. What we can do, however, is to utilize existing components that already built this functionality. Let us say that we want to embed a Boo's code editor in our application. Here is what we need to do:

public class TextEditorForm : Form
{
    public TextEditorForm()
    {
        var editorControl = new TextEditorControl
        {
            Dock = DockStyle.Fill
        };
        Controls.Add(editorControl);
        editorControl.Document.FormattingStrategy = new BooFormattingStrategy();
        editorControl.SetHighlighting("Boo");
    }
}
public class BooFormattingStrategy : DefaultFormattingStrategy
{
    public override void IndentLines(TextArea textArea, int begin, int end)
    {
    }

    protected override int SmartIndentLine(TextArea area, int line)
    {
        IDocument document = area.Document;
        LineSegment lineSegment = document.GetLineSegment(line - 1);
        if (document.GetText(lineSegment).EndsWith(":"))
        {
            LineSegment segment = document.GetLineSegment(line);
            string str = base.GetIndentation(area, line - 1) + Tab.GetIndentationString(document);
            document.Replace(segment.Offset, segment.Length, str + document.GetText(segment));
            return str.Length;
        }
        return base.SmartIndentLine(area, line);
    }
}

And that gives us this:

Nice :-)

Before getting to the more complex scenarios of creating professional DSL, I wanted to start by showing how you can easily create your own syntax highlighting.

This was the result:

And this is the actual code that makes this happens:

private void codeTextBox_TextChanged(object sender, EventArgs e)
{
    int prevSelectionStart = codeTextBox.SelectionStart;
    int prevSelectionLength = codeTextBox.SelectionLength;
    codeTextBox.SelectionStart = 0;
    codeTextBox.SelectionLength = codeTextBox.TextLength;
    codeTextBox.SelectionColor = DefaultForeColor;

    var keyWords = new[] { "specification", "requires", "users_per_machine", "same_machine_as" };
    foreach (string keyWord in keyWords)
    {
        MatchCollection matches = Regex.Matches(codeTextBox.Text, keyWord);
        foreach (Match match in matches)
        {
            codeTextBox.SelectionStart = match.Index;
            codeTextBox.SelectionLength = match.Length;
            codeTextBox.SelectionColor = Color.DarkOrchid;
        }
    }
    foreach (Match match in Regex.Matches(codeTextBox.Text, @"@[\w\d_]+"))
    {
        codeTextBox.SelectionStart = match.Index;
        codeTextBox.SelectionLength = match.Length;
        codeTextBox.SelectionColor = Color.DarkSeaGreen;
    }
    foreach (Match match in Regex.Matches(codeTextBox.Text, @" \d+"))
    {
        codeTextBox.SelectionStart = match.Index;
        codeTextBox.SelectionLength = match.Length;
        codeTextBox.SelectionColor = Color.DarkRed;
    }

    codeTextBox.SelectionStart = prevSelectionStart;
    codeTextBox.SelectionLength = prevSelectionLength;
}

The code in the book comes with the following warning:

The code suffers from multiple bugs, issues and is generally not suited for anything but the simplest scenarios.

I am currently writing the chapter about creating professional DSL. One of the subjects that I am dealing with is creating graphical representations of a textual DSL.

Here is the DSL script in question:

specification @vacations:
	requires @scheduling_work
	requires @external_connections
	
specification @salary:
	users_per_machine 150
	
specification @taxes:
	users_per_machine 50

specification @pension:
	same_machine_as @health_insurance

I am not sure what kind of UI representation this should have.

For that matter, let us take the example that I commonly use, of a rule engine for an ordering system:

when order.total > 1000 and order.payment_method = 'CreditCard':
	apply_discout_of 5.precent

How would you visualize that?

A few days ago I posted about Ugly Linq. Ever since then, I kept thinking about how ugly it is to handle this by hand. Suddenly, it hit me that I don't have to do it that way.

Boo already has the facilities to take a compiler AST and translate that into the code that would recreate this AST. In particular, this makes the code we previously had to write to this:

public class ConditionMacro : AbstractAstMacro
{
	public override Statement Expand(MacroStatement macro)
	{
		Expression serialize = new CodeSerializer().Serialize(macro.Arguments[0]);
		var body = new Block();
		body.Statements.Add(new ReturnStatement(macro.Arguments[0]));
		return new ExpressionStatement(
			new MethodInvocationExpression(
					AstUtil.CreateReferenceExpression(typeof(Condition).FullName),
					new BlockExpression(body),
					serialize
				)

			);
	}
}

And what that means is that given this code:

condition a > 10

We can get this result:

And that is it.

You get both the actual compiled expression and the AST that describes this. This is critically important because you can now take this piece of AST and do transformations / views on it.

And that is important if you want to have reliable graphical representation on top of a textual DSL, which is what my chapter 10 is going to cover.

Damn, this is simple! Thanks Rodrigo!

One of the common misconceptions about a DSL is that it to think about each DSL script independently. As a simple example, let us go back to the routing DSL, we may have a rule like this:

priority 10
when msg is NewOrder and msg.Amount > 10000:
    dispatch_to "orders@strategic" 

And our focus is mainly on the actual language and syntax that we want to get. This is a mistake, because we aren't considering the environment in which the DSL lives. In the same sense that we rarely consider a code file independently, we should not consider each script independently.

With the routing DSL (and yes, I am stretching the example), we may need to perform additional actions, rather than just dispatch the message. For example, we may want to log all strategic messages. As you can see, we can easily add this to the DSL:

priority 10
when msg is NewOrder and msg.Amount > 10000:
     log "strategic messages" , msg
     dispatch_to "orders@strategic"

However, this is a violation of SoC, and we care about such things with a DSL just as much as we care about them with code. So we can do it like this, leave the original snippet and add another, like this:

when destination.Contains("strategic"):
	log "strategic messages", msg

Now the behavior of the system is split across several files, and it is the responsibility of the DSL engine to deal with this appropriately. One way to arrange this would be this folder structure:

• /routing_scripts
• /routes
• /orders
• dispatch_big_new_orders_to_strategic_customers_handling
• dispatch_standard_new_orders_to_normal_customers_handling
• /behaviors
• /after
• /log_strategic_messages
• /dispatch_to_error_queue_if_not_dispatched

The DSL engine can tell (from the message) that it needs to execute only the routing rules in /routes/orders, and it can execute the before and after actions without getting the routing scripts tied to different concerns.

If you want to be a stickler, we are actually dealing with two dialects that are bound to the same DSL engine. In this case, they are very similar to one another, but that doesn't have to be the case.

Multi file DSL doesn't have to be just about combining different DSLs together, they are also about binding different scripts. Let us look at another possible folder structure:

/routing_scripts

• /routes
• /orders
• dispatch_big_new_orders_to_strategic_customers_handling
• convention_based_dispatching
• dispatch_to_error_if_not_dispatched

In this case, the DSL is based around the idea of executing things in reverse depth order. That is, when a message arrives, we try to match it to the deepest scope possible (in this case, handling strategic customers), and we go up until we reach the root.

This is still, however, just another way of bringing different scripts together. Albeit in a fairly interesting ways.

Let us consider a completely different example, order management. We might have something like this:

// this is part of the rules for processing an order
when order.Total > 1000:
	large_order

// this is the large_order script
order.TaxRate = 0.10
if user.Location in (complex_taxation.Locations):
	complex_taxation.Handle(order)

// this is the complex_taxation/mars script
if amount % 13 == 0:
	return 0.0
return amount * 0.04

Admittedly not the best example as far as business logic goes, but you can see the interaction of three different scripts and how they rely on one another to split the complexity of the system up. It also means that changing the taxation rules on mars is not going to touch how we are going to handle an order, which is good (SoC again).

In this case, I don't think that there is any doubt that those scripts are truly part of a multi file DSL.

Again, when designing and building a DSL, thinking about just a single file is the easiest way to go, but you ought to consider the entire environment when you make such decisions. A language that does not support SoC is bound to get extremely brittle very fast.

One of the common mistakes in a DSL is to create one that solve a demo problem and leave it at that.

As a simple example, let us say that we build a DSL to handle order management. Here is a typical scenario that was talked about during development:

when is_preferred_user and order_amount > 500:
	apply_discount 5.precent

Sounds reasonable, right?

Except, that this isn't really a good way to express a business rule. A common business rule usually have a lot more complexity to it. Here is a good example:

when user.payment_method is credit_card and ((order_amount > 500 and order_amount < 1200) or number_of_payments < 4) and 
    user.is_member_of("weekend buy club") and Now.DayOfWeek in (DayOfWeek.Sunday, DayOfWeek.Sutarday) 
    and applied_discounts < 10:
	apply_discount 5.precent

At a glance, what the !@$!# does this rule do?

This is a good example of stagnant DSL. It is no longer being actively developed (easily seen by the use of framework terms such as DayOfWeek), and the complexity is growing unchecked.

This is usually the case when the DSL design has not taken into account new functionality, or when it relies on developers to be able to extend the language when needed. Because it requires developer involvement, it is often easier to patch it by complex conditional rather than express the actual business logic in a readable way.

A good way to avoid that is to incorporate known best practices from the development side into our DSL. In other words, we need to allow encapsulation and extensibility in our DSLs. As a simple example, we can allow the following:

condition weekend_club_member:
	user.is_member_of("weekend club memeber")
condition sale_on_weekend:
	Now.DayOfWeek in (DayOfWeek.Sunday, DayOfWeek.Sutrday)
condition good_payment_option: # yes, I know, bad name, deal with it
	((order_amount > 500 and order_amount < 1200) or number_of_payments < 4)

And now the condition becomes:

when user.payment_method is credict_card and good_payment_option and sale_on_weekend 
     and weekend_club_member and applied_discounts < 10:
	apply_discount 5.precent

This is important, because it avoid a language rot and provide the facilities to the end user to add abstraction levels.

Previously on the Multi Tenancy series:

• Multi Tenancy - Definition and scenario.
• Reviewing Litware HR - P&P Multi Tenancy Sample.
• Multi Tenancy - The Physical Data Model
• Multi Tenancy - Extensible Data Model
• Multi Tenancy - Extensible Behaviors
• Multi Tenancy - Development Structure
• Multi Tenancy - Where do you put variability?

It probably would not surprise you that I a seeing a place for a DSL in a Multi Tenant application. Indeed, one might almost say that I have DSL on my mind a lot these days.

Multi Tenant applications are all about managing variability between the tenants. A sweet spot for a DSL implementation is to manage policy while the application code define the actual operations.

Therefor, it shouldn't come as a surprise that in many instances, where the variability in the application between one tenant and another is entirely (or predominantly) in policy, as it often is, a DSL is a good solution. In particular, defining such things as workflows, attaching behaviors to events in the applications, defining rules, etc. As you can imagine, this is not a single DSL, but a set of languages, each used for a particular task.

I am fond of this approach, for the obvious reasons, but also because it is the easiest way to bring the tenant customization into the tenants hands directly. If you go with that route, you need to be careful about managing the current state (to allow rolling back a changed script) and to ensure that you are isolating the actual application from the client's code, which may very well be written in such a way that would harm the application. As such, it require a bit more investment in the infrastructure than letting developers handle things, but you get a large benefit out of it.

I asked what is wrong with the following DSL:

specification @vacations:
    requires @scheduling_work
    requires @external_connections

I also constrained the problem so it can't be any of the following:

• It has nothing to do with the implementation.
• It has nothing to do with the actual engine running this.
• Look for what isn't there.
• It is only a problem when you scale up.
• And no, there are no performance problems whatsoever.

The problem has nothing to do with technical issues, and all to do with management issues. The code above makes a certain modification to the current state of the system, but it doesn't tell us why!

Consider the case where we have several hundreds (or thousands) of those rules.  How are you going to associate the end result of executing all those rules with the actual rule that caused a particular issue? How are you going to know why?

For a well defined DSL, we need more than just the syntax, we need an environment that supports the whole development process, and that includes being able to give good answers during runtime, about what caused what and why. Some of that are things that you can add to the syntax, that is, you can write is as:

specification @vacations: 
	requires @scheduling_work, "Vacations requires work scheduling so we can plan around vacations"
	requires @external_connections, "We need to order vacation spots"

Note: I don't want to hear about the ridiculousness of the business example, I know it is. It is here as an example for something else, not the actual business logic.

But this still doesn't give you enough information, the DSL engine itself needs to record the information about what rule (and what script) had modified the state of the application, and allow you to just directly from the result to the rule that allowed it.

Otherwise, what you have is chaos.

An incomplete list of subjects for the 10th chapter.

The editor environment

• Color highlighting
• Intellisense
• Sharp Develop
• Visual Studio

DSL Code Generation

• The Code DOM Provider for Boo
• Specific DSL Writers

Graphical UI

• Displaying the DSL
• Saving DSL modifications
• The value of a graphical front end

Errors and warnings handling

Debugging

Dealing with the large scale - what the hell is going on?

At long last, I am done with Chapter 9. The actual writing process didn't took so long, but the ideas took a long time to come into a coherent form.

Here is the final table of content:

9.1 Starting from a stable origin
9.2 Planning our DSL's versioning story
9.2.1 Implications of modifying the DSL engine
9.2.2 Implications of modifying the DSL API and Model
9.2.3 Implications of modifying the DSL Syntax
9.2.4 Implications of modifying the DSL Environment
9.3 Regression Test Suite
9.4 Versioning Strategies
9.4.1 Abandon Ship Strategy
9.4.2 Single Shot Strategy
9.4.3 Additive Change Strategy
9.4.4 Tower of Babel Strategy
9.4.5 Adapter / Compatibility Mode Strategy
9.4.6 The Great Migration Strategy
9.5 Applying versioning strategies
9.5.1 - Managing safe, additive, changes
9.5.2 Handling required breaking change
9.6 DSL versioning in the real world
9.6.1 Versioning Brail - Boo based templating language
9.6.2 Versioning Binsor - Inversion of Control Container Configuration DSL
9.6.3 Versioning Rhino ETL - Extract Transform Load DSL
9.7 Versioning - only at release boundary
9.8 Summary

And the chapter summary:

In this chapter, we have discussed how we can ensure the longevity of our languages, how we can make them survive and prosper from one version to the next.

Of particular importance are the tests and the regression test suite, which we use to ensure that existing behavior is not changed in an incompatible way, those breaking client scripts.

We also discussed the implication of versioning on the design and implementation of our DSLs. In particular, we mentioned the importance of using façades to separate our DSL from the application's model and API, in order to allow them both to be developed independently. This separation goes a bit further than merely a façade, however. We want to ensure that the DSL engine is usable even without the DSL.

This seems like a strange requirement, but it is a very important one. You want to be able to modify the engine without modifying the DSL, and vice versa. In addition to that, you might want to have several dialects of the same language (typically several versions) based on the same engine. This tends to be hard to impossible if there isn't a good level of isolation between the two.

The DSL environment (evaluation order, invocation call sites and naming conventions)and its implication on versioning are something that most people tend to forget about, until they find out just how critical it is for the usage of the language.

We walked through several versioning strategies, from the "we always start fresh" to the "no changes are allowed" to multi dialect languages and automatic migration tools. We also applied a few of those strategies to the Quote Generation DSL, and then seen how real world DSLs have dealt with the problem, and the various constraints that led to the different choices that were made.

Versioning is a big topic, and there is not enough room in a single chapter to cover it all. Most of the guidance about versioning is directly applicable for languages as well, and I strongly recommend referring to it as well as this chapter.

At this point, we are surely ready to release our DSL to the whole wide world, right? Well, from the technical perspective, certainly. But we are still missing something, a bit of shine, just a hint of polish.

Creating a DSL is easy, making is slick and professional is another matter, that is what we will spend the next chapter discussing.

Next chapter - Creating professional DSL.