This code is part of a DSL that is used to generate quotes. In particular, this bit is used to build the dependency graph for the engine to run on.

specification @vacations:
	requires @scheduling_work
	requires @external_connections

There is a serious design issue with this bit of code. Can you figure it out?

Hints:

• It has nothing to do with the implementation.
• It has nothing to do with the actual engine running this.
• Look for what isn't there.
• It is only a problem when you scale up.
• And no, there are no performance problems whatsoever.

I am working  on the versioning chapter for the book, and I am current at the place where I am suggesting letting the compiler know about what kind of situations your users are likely to get into.

The examples I am using is changing the API from:

requires @vacations

To:

requires @vacations, "Some explanatory text"

And handling the scenario where the user attempts to write to the old API.

Here is the default approach:

BCE0017: Boo.Lang.Compiler.CompilerError: The best overload for the method 'BDSLiB.QuoteGeneration.QuoteGeneratorRule.requires(string, string)' is not compatible with the argument list '(string)'.

We can use the standard [Obsolete] mechanism, like this:

[Obsolete("use requires(moduleName, explanation) instead", true)]
public void requires(string moduleName)
{
	throw new NotSupportedException();
}

And the error would be:

'BDSLiB.QuoteGeneration.QuoteGeneratorRule.requires(string)' is obsolete. use requires(moduleName, explanation) instead

This is still very scary message for non technical users. So we can up the cost a bit using this:

[Meta]
public static Expression requires(Expression moduleName)
{
	var message =
		@"
Requiring a module without supplying an explanation is not allowed. 
Please use the following syntax:
'requires " + moduleName + "', '" + moduleName + " is required because [add your reasoning here]'";
	CompilerContext.Current.Errors.Add(new CompilerError(moduleName.LexicalInfo, message));
	return new MethodInvocationExpression
	{
		Target = new ReferenceExpression("requires"),
		Arguments = new ExpressionCollection
		{
			moduleName,
			new StringLiteralExpression("No explanation specified")
		}
	};
}

In which case the error is:

BCE0000: Boo.Lang.Compiler.CompilerError:
Requiring a module without supplying an explanation is not allowed.
Please use the following syntax:
'requires 'scheduling_work'', ''scheduling_work' is required because [add your reasoning here]'

This is a much nicer message to get, I think you would agree.

Thoughts?

• Starting from a stable origin
• Planning our DSL's versioning story
• Implications of modifying the DSL Engine
• Implications of modifying the DSL API and Model
• Implications of modifying the DSL Syntax
• Implications of modifying the DSL Environment
• Regression Test Suite
• Versioning cut off point - where to apply versioning concerns
• Versioning strategies:
• Abandon Ship Strategy
• Glacial Change Strategy
• Additive Change Strategy
• Tower of Babel Strategy
• Runtime Adapter Strategy
• Compilation Adapter Strategy
• Auto Modification Strategy
• Self Modification Strategy
• Versioning best practices:
  # actual content
• Closed world
• Regression test suite

The title says it all, I think. Okay, it doesn't, I admit.

There is a lot of focus with DSL about the syntax. And there is some focus on the engine. There is very little focus about how both the environment and the usage of the DSL affect the DSL itself. Here are a few examples:

• Naming convention
• Script ordering
• Execution location (when you are running the scripts)

All of those are of particular importance in many DSL, not only for the actual execution, but for how the DSL itself is written.

Those are just a few topics that I feel are important for discussion when talking about versioning DSL:

• Different behavior at runtime
• API vs. Syntax
• Different dialects
• Backward and forward Compatibility
• Pros:
• Keeping existing assets
• Training
• Knowledge
• The Test of Fire
• Cons:
• Increased costs
• Harder to change
• Preparing for versioning:
• Syntax Documentation
• Closed world - control what you can access
• Limit to a scenario
• Versioning strategies:
• The Holy Compatibility
• Build & abandon
• Version marker
• The Big Upgrade

Thoughts?

Nathan Stott is doing some really interesting things with Rhino DSL and Boo. His latest post outlines how to create this syntax:

CreateTable "Cats":
    Int32 "Id", { "identity" : true, "primary" : true }
    String "Name", { "length" : 50 }

I like it.

Nathan has posted Simple State Machine to CodePlex, it is the first project that I am aware of that uses Rhino DSL and the techniques that I am talking about in the book.

What is impressive about this is the level of professionalism that is involved in the project. It is a full scale DSL, with all the supporting infrastructure. I spent half an hour or so going through the entire thing, and I am impressed.

Put simply, this is how I think state based work flows should be defined. I could easily see myself extending this a bit to add persistence support & integration with NServiceBus, and be done with it.

Like most state machines, it has the ideas of states, events that can cause the state to be changed, and legal transitions from state to state. You can define tasks which will be executed upon changing a state, or upon entering / leaving a certain state.

Enough talking, let us look at a reasonably complex work flow:

workflow "Order Lifecycle"

#Event & State Identifier Targets.
#This section controls which Types will be used
#to resolve Event or State names into strongly typed CLR objects.
#--------------------------------------------------------
state_identifier_target @OrderStatus
event_identifier_target @OrderEvents

#Global Actions
#--------------------------------------------------------
on_change_state      @WriteToHistory, "on_change_state"
on_workflow_start    @WriteToHistory, "on_workflow_start"
on_workflow_complete @WriteToHistory, "on_workflow_complete"

#Event Definitions
#--------------------------------------------------------
define_event  @OrderPlaced
define_event  @CreditCardApproved
define_event  @CreditCardDenied
define_event  @OrderCancelledByCustomer
define_event  @OutOfStock
define_event  @OrderStocked
define_event  @OrderShipped
define_event  @OrderReceived
define_event  @OrderLost

#State & Transition Definitions
#--------------------------------------------------------
state @AwaitingOrder:
       when @OrderPlaced              >> @AwaitingPayment

state @AwaitingPayment:
       when @CreditCardApproved       >> @AwaitingShipment
       when @CreditCardDenied         >> @OrderCancelled
       when @OrderCancelledByCustomer >> @OrderCancelled

state @AwaitingShipment:
       when @OrderCancelledByCustomer >> @OrderCancelled
       when @OutOfStock               >> @OnBackorder
       when @OrderShipped             >> @InTransit

       #Individual states can define transition events as well
       on_enter_state @WriteToHistory, "on_enter_state(AwaitingShipment)"

state @OnBackorder:
       when @OrderCancelledByCustomer >> @OrderCancelled
       when @OrderStocked             >> @AwaitingShipment

state @InTransit:
       when @OrderReceived            >> @OrderComplete
       when @OrderLost                >> @AwaitingShipment

#NOTE: State definitions without any transitions will cause
#the state machine to Complete when they are reached.
#------------------------------------------------------------
state @OrderComplete
state @OrderCancelled

Here is the demo application UI, for the order processing life cycle:

As I said, impressive.

I am considering having a language that mandates tests. If you don't have a matching test for the code in question, it will refuse to run. If the tests fail, it will refuse to run. If the tests takes too long, they are considered failed and the code will refuse to run.

This certainly ensure that there would be test. It wouldn't ensure that they would be meaningful, however. That is fine by me. I am not interested in policy through enforcement, just gentle encouragement in the right direction.

The technical challenges of implementing such a system are nil. The implications on the workflow and ease of use for such a system are unknown. On the surface, checked exceptions are great. In practice, they are very cumbersome. This is why I am warning that I have only toyed with the idea, not implemented it.

Thoughts?

when I am writing DSL, I keep hitting one pain point. The CLR naming conventions, which are more or less imprinted on my eyelids, are not really conductive to clear reading in a DSL.

Let us take these entities, and see what we get when we try to build a DSL from them:

The DSL is for defining business rules, and it looks like this:

when User.IsPreferred and Order.TotalCost > 1000:
	AddDiscountPrecentage  5
	ApplyFreeShipping
when not User.IsPreferred and Order.TotalCost > 1000:
	SuggestUpgradeToPreferred 
	ApplyFreeShipping
when User.IsNotPreferred and Order.TotalCost > 500:
	ApplyFreeShipping

The main problem with this style of writing is that it is visually condense. I can read it pretty much as easily as I read natural English, but anyone who is not a developer really have to make an effort, and even for me, trying to read ruby styled code is easier. Here is how this would look like when using the ruby style conventions:

when User.is_preferred and Order.total_cost > 1000:
    add_discount_precentage 5
    apply_free_shipping
when not User.is_preferred and Order.total_cost > 1000:
   suggest_upgrade_to_preferred 
    apply_free_shipping
when User.is_not_preferred and Order.total_cost > 500:
   apply_free_shipping

This is much easier to read, in my opinion. The problem is that I consider this extremely ugly.

Obviously a different solution is needed...

Wait a minute! Boo has an open compiler. Why not just change the way it handle references? And that is what I did:

///<summary>
/// Allow to use underscore separated names, which will be translated to pascal case names.
/// pascal_case -> PascalCase.
/// All names that contains an underscores will go through this treatment.
///</summary>
/// <example>
/// You can  enable this behavior using the following statement
/// <code>
/// compiler.Parameters.Pipeline
///		.Replace(typeof (ProcessMethodBodiesWithDuckTyping),
/// 				 new ProcessMethodBodiesWithDslNamesAndDuckTyping());
/// </code>
/// </example>
public class ProcessMethodBodiesWithDslNamesAndDuckTyping : ProcessMethodBodiesWithDuckTyping
{
	/// <summary>
	/// Called when we encounter a reference expression
	/// </summary>
	/// <param name="node">The node.</param>
	public override void OnReferenceExpression(ReferenceExpression node)
	{
		if(node.Name.Contains("_"))
			SetNodeNameToPascalCase(node);
		base.OnReferenceExpression(node);
	}

	/// <summary>
	/// Called when we encounters a member reference expression
	/// </summary>
	/// <param name="node">The node.</param>
	public override void OnMemberReferenceExpression(MemberReferenceExpression node)
	{
		if (node.Name.Contains("_"))
			SetNodeNameToPascalCase(node);
		base.OnMemberReferenceExpression(node);
	}

	/// <summary>
	/// Sets the node name to pascal case.
	/// </summary>
	/// <param name="node">The node.</param>
	private static void SetNodeNameToPascalCase(ReferenceExpression node)
	{
		string[] parts = node.Name.Split(new char[] { '_' },StringSplitOptions.RemoveEmptyEntries);
		StringBuilder name = new StringBuilder();
		foreach (var part in parts)
		{
			name.Append(char.ToUpperInvariant(part[0]))
				.Append(part.Substring(1));
		}
		node.Name = name.ToString();
	}
}

I love Boo, with cause.

Roughly speaking, a DSL is composed of the following parts:

It should come as no surprise that when we test it, we test each of those components individually. When the time comes to test a DSL, I have the following tests:

• CanCompile - This is the most trivial test, it assert that I can take a known script and compile it.
• Syntax tests - Didn’t we just test that when we wrote the CanCompile() test? When I am talking about testing the syntax I am not talking about just verifying that it can compile successfully. I am talking about whatever the syntax that we have created has been compiled into the correct output. The CanCompile() test is only the first step in that direction. Here is an example of such a test.
• DSL API tests - What exactly is the DSL API? In general, I think about the DSL API as any API that is directly exposed to the DSL. The methods and properties of the anonymous base class is an obvious candidate, of course. Anything else that was purposefully built to be used by the DSL also fall into this category. Those I test using standard unit tests, without involving the DSL at all. Testing in isolation again.
• Engine tests - A DSL engine is the responsible for managing the interactions between the application and the DSL scripts. It is the gateway to the DSL in our application, allowing us to shell out policy decisions and oft-changed rules to an external entity. Since the engine is usually just a consumer of the DSL instances, we have several choices when the time comes to create test cases for the engine. We can perform a cross cutting test, which would involve the actual DSL, or test just the interaction of the engine with the provided instances. Since we generally want to test the engine behavior in invalid scenarios (a DSL script which cannot be compiled, for example), I tend to choose the first approach.

Testing the scripts

We have talked about how we can create tests for our DSL implementation, but we still haven’t talked about how we can actually test the DSL scripts themselves. Considering the typical scenarios for using a DSL (providing a policy, defining rules, making decisions, driving the application, etc), I don’t think anyone can argue against the need to have tests in place to verify that we actually do what we think we do.

In fact, because we usually use DSL as a way to define high level application behavior, there is an absolute need to be aware of what it is doing, and protect ourselves from accidental changes.

One of the more important things to remember when dealing with Boo based DSL is that the output of those DSL is just IL. This means that this output is subject to all the standard advantages and disadvantages of all other IL based languages.In this specific case, it means that we can just reference the resulting assembly and perform something write a test case directly against it.

In most cases, however, we can safely utilize the anonymous base class as a way to test the behavior of the scripts that we build. This allows us to have a nearly no-cost approach to building our tests. Let us see how we can test this piece of code:

specification @vacations:
	requires @scheduling_work
	requires @external_connections

specification @scheduling_work:
	return # doesn't require anything

And we can test this with this code:

[Test]
public void WhenUsingVacations_SchedulingWork_And_ExternalConnections_AreRequired()
{
	QuoteGeneratorRule rule = dslFactory.Create<QuoteGeneratorRule>(
		@"Quotes/simple.boo",
		new RequirementsInformation(200, "vacations"));
	rule.Evaluate();

	SystemModule module = rule.Modules[0];
	Assert.AreEqual("vacations", module.Name);
	Assert.AreEqual(2, module.Requirements.Count);
	Assert.AreEqual("scheduling_work", module.Requirements[0]);
	Assert.AreEqual("external_connections", module.Requirements[1]);
}

Or we can utilize a test DSL to do the same:

script "quotes/simple.boo"

with @vacations:
	should_require @scheduling_work
	should_require @external_connections	

with @scheduling_work:
	should_have_no_requirements

Note that creating a test DSL is only worth it if you expect to have a large number of DSL scripts of the tested language that you want to test.