Domain Specific Languages
DotNetRocks on Domain Specific Languages
Last week I recorded a DNR session about Domain Specific Languages. I also talked just a bit about writing your own DSL, some challenges that I run into since the book, the Boo language and why it is suitable for DSLs and how the entire process works. Looking forward for your comments.
DSLs in Boo discount offer
Manning is running a special today for my book, 50% off print & ebook versions. Just use the following code: boo50 Note that this code is valid only for today, so hurry up!
DSLs in Boo is out!
It has been quite a journey for me, starting in 2007(!) up until about a month ago, when the final revision is out. I am very happy to announce that my book is now available in its final form. When I actually got the book in my hands I was ecstatic. That represent about two years worth of work, and some pretty tough hurdles to cross (think about the challenge that editing something the size of a book from my English is). And getting the content right was even harder. On the one hand, I wanted...
My book is on Manning – deal of the day
My book is on Manning’s Deal of the Day. DSLs in Boo: Domain Specific Languages in .NET – get the ebook for $10 by using this link. The journey for the book is done, so it is the full deal and the final version that you’ll be getting.
Help needed: Writing Domain Specific Languages in Boo – Java Edition
Just came out of a discussion with Manning about my book (which is approach the last few stages before actually printing!), apparently they hit upon the notion that Boo works on both the CLR and the JVM, and are interested in having a Java edition of the book. Disclaimer: This is early, and anything is subject to change, blah blah blah. I find this hilarious, since this is Hibernate in Action vs. NHibernate in Action, in reverse. At least, I hope it is. The problem? I am not familiar enough with Java to be...
M is to DSL as Drag & Drop is to programming
I have been sitting on this post for a while now, because that was my first impression after seeing Oslo & M and all the hype around it from the PDC. To be frank, I had a hard time believing my own gut feeling. I kept having the feeling that I am missing something, which is why I avoided talking about this so far. But, as time passed, and as we started to see more and more about Oslo and M, it validated my initial thinking. Now, just to be clear, I don’t intend to even...
Building Domain Specific Languages with Boo – Full book now available
Building Domain Specific Languages with Boo Another big milestone, early access subscribers can now read the entire book, all 13 chapters and both appendixes. Not edited yet, but much closer to completion.
DSL: Tests as documentation
I have several DSL that have no documentation beyond their source and the tests. They are usable, useful and have been of a lot of help. However, I have run into situations where I, as the language author, could not answer a question about the language without referring to the code. I strongly recommend in investing the time to create good documentation in your DSL. Even if you are using a Behavior Driven Design flavored tests, it is not quite enough. Those types of tests can help make it clear what the language is doing, but they are not the...
I need some advice about the book
I have got the final reviews about the book (Building Domain Specific Languages with Boo), and a few of them talk about a problem in the book that I am not sure how to solve. The problem is, quite simply, the book talks about Boo a lot. I see the book not as a academic topic, but as something that would give the reader actionable knowledge. At the same time, I try to cover the entire life cycle of a DSL. So the book covers things like how to design a good syntax, documenting DSLs, how to structure them,...
Should you be able to define new abstractions in a Domain Specific Language?
Fowler has a post about DSL, which contain the following: The first is in language design. I was talking with a language designer who I have a lot of respect for, and he stressed that a key feature of languages was the ability to define new abstractions. With DSLs I don't think this is the case. In most DSLs the DSL chooses the abstraction you work with, if you want different abstractions you use a different DSL (or maybe extend the DSL you're using). Sometimes there's a role for new abstractions, but those cases are the minority and when...
A very significant milestone
Today, around 3 AM, I finally submitted the last chapter in the book. I was surprised to realize that it took over a year to write it, and I am happy beyond words that the action of writing the book is behind me. That said, it is not actually the end of the work. All it means is that I am done writing new content, now we are going to get into a cycle of editing once more, and I really don't look forward to re-reading what I wrote five to ten times, over and over and over again. Nevertheless,...
Integrating User Interface & Domain Specific Languages
User interface is often seen as an external piece of the DSL, something that is completely separated from the language. While this is one options to handle things, it is not necessarily the only one. During the design of the DSL, we should also think about the user interface that we intend to give our users. That will change the way that we build the language, usually, but not only because we need to give the UI more information, but because we can also utilize the UI to make things easier. In particular, a good usage of the UI will...
The shopping cart rule engine DSL
As the final example in the book, I am showing off a DSL for processing a shopping cart. Here is how it looks like: behavior of preferred_customer
upon bad_credit:
authorize_funds cart.Total * 0.5, "For preferred customers we only authorize half the amount"
upon cart_update:
when cart.Total > 1000:
add_cart_discount 5, "Preferred members gets 5% discount for orders over 1,000$"
And this:
behavior of default_customer
upon bad_credit: authorize_funds cart.Total, "We require full authorization of the amount in low credit rating scenarios"
Where we defined preferred customer and default customer as:
define preferred_customer:
customer.TotalPurchaseAmount > 5000
define default_customer:
customer.TotalPurchaseAmount <= 5000
The implementation is surprisingly easy, and I was able to walk...
Kazien Conf Workshops
Yesterday I gave two workshops, Advanced NHibernate and Building DSL with Boo. I finished the day absolutely crushed, but I think they went very well.
Both were recorded, although I am not sure when they will be online.
Ryan Kelley has a blow by blow description of the NHibernate talk, and you can get the code for that here:
https://rhino-tools.svn.sourceforge.net/svnroot/rhino-tools/trunk/SampleApplications/ORM+=2/
I'll post the code for the DSL talk shortly afterward.
I got some really positive feedback about the NHibernate Profiler, and I am very interested in demoing that and getting additional feedback when the real conference starts.
Every DSL ends up being Smalltalk
I had this though in my head for a while now. I built an IDE for a DSL, and somewhere toward the end of the first revision I understood something very interesting. My IDE wasn't actually using the textual representation of the language. The scripts that the user was editing were actually live instances, and they were fully capable of validating and saving themselves. The IDE worked with those instances, used them to do its operations, and allowed to edit them on the fly. It was quite challenging to do, I must say, and I kept thinking about the image...
New version of the book is out
You can head to the Book Site and get a new version, because Chapters 1 - 11 (and appendix B) are available.
The only remaining chapters are #12 (DSL implementation patterns) and #13 Implementing a Real World DSL. Chapter 12 goes really well, so far, and I hope to have it ready by the end of the weekend.
Chapter 11 is done, or the tale of meta documentation
Here it the table of content: Writing the Getting Started Guide Create Low Hanging Fruits The User Guide Documenting the Language Syntax The Language Reference Debugging for business users Creating developer documentation Outlining DSL structure The syntax implementation Keywords Behaviors Conventions Notations External Integration Documenting AST transformations Executable documentation Summary The chapter starts with... Documentation is a task that most developers strongly dislike. It is treated as a tedious, annoying task and often falls on the developer who protests the least. One additional problem is that a developer trying to document his own work...
Enter the demoware
I am writing about documentation at the moment, and I found myself writing the following: I don’t think that I can emphasize enough how important it is to have a good first impression in the DSL. It can literally make or break your project. We should make above reasonable efforts to ensure that the first impression of the user from our system would be positive. This includes investing time in building good looking UI, and snappy graphics. They might not actually have a lot of value for the project from a technical perspective, not even from the point...
Persistent DSL caching issues
A while ago I talked about persistent DSL caching. I was asked why my solution was not a builtin part of Rhino DSL. The reason for that is that this is actually a not so simple problem. Let me point out a few of the issues that are non obvious. Need to handle removal of scripts Need to handle updating scripts Need to handle new scripts Those are easy, sort of, but what about this one? Need to handle DSL updates ...
Implementing generic natural language DSL
I said that I would post about it, so here is the high level design for generic implementation of natural language looking parsing. Let us explore the problem scenario first. We want to be able to build this language, without having to build a full blown language from scratch: open http://www.ayende.com/ click on link to Blog click on link to first post enter comment with name Ayende Rahien and email foo@example.org and url http://www.ayende.com/Blog/ enter comment...
The search for the natural language
Jeremy Miller is looking for a DSL that reads like natural language. My immediate response was that it is not practical, because I assumed he wanted very natural language, which is still not possible to do without extremely high budget. Limiting the problem to just reads like a natural language reduce the problem space significantly. I am going to have a separate post about how to actually solve such a problem, but for now, I want to talk about the actual requested solution. I think it is 100%solvable with a low cost approach. That is, you can get a...
Answer: Don't stop with the first DSL abstraction
The problem as it was stated was of rules that looked like this: upon bounced_check or refused_credit:
if customer.TotalPurchases > 10000: # preferred
ask_authorization_for_more_credit
else:
call_the cops
upon new_order:
if customer.TotalPurchases > 10000: # preferred
apply_discount 5.precent
upon order_shipped:
send_marketing_stuff unless customer.RequestedNoSpam
I don't like it, and the reason isn't just that we can introduce IsPreferred.
I don't like it because the abstraction facilities here are poor. We have basically introduced events and business rules, maybe with a sprinkling of a domain model, but nothing really meaningful. Such system will die under their own weight in any situation of significant complexity (in other words,...
Challenge: Don't stop with the first DSL abstraction
I was having a discussion today about the way business rules are implemented. And large part of the discussion was focused on trying to get a specific behavior in a specific circumstance. As usual, I am going to use a totally different example, which might not be as brutal in its focus as the real one. We have a set of business rules that relate to what is going to happen to a customer in certain situations. For example, we might have the following: upon bounced_check or refused_credit:
if customer.TotalPurchases > 10000: # preferred
ask_authorizatin_for_more_credit
else:
call_the cops
upon new_order:
if...
Persistent DSL Caching
This is a note to myself, because I don't have the time for a proper post. When you are dealing with a DSL that contains more than just a few scripts, you really being to care about compilation times. Even with caching, this can be a problem. The solution is the same that we have been using for the last three to four decades, don't compile if the source hasn't changed. The code to make this happen using Rhino DSL is here: public override CompilerContext Compile(string[] urls)
{
var outputAssemblyName = OutputAssemblyName(urls);
...
Chapter 11 - Documenting your DSL
I am going to start writing this soon. Any suggestions? As a bait, I am going to apply the techniques that I intend to describe to Binsor, which should give you incentive to say what you really want from a DSL documentation... :-)
Chapter 10 is done, happy
A few hours ago I finished writing Chapter 10. Looking at the time on the calendar, it didn't take too long, about a month. Looking at it from the point of view of effort involved, I feel about as tired as if it took three years. Other chapters took longer, but didn't take out quite as much effort. Not sure what this means... but I am happy I am done with it (for a given value of done, I will still need to review and read it at least three dozen times).
A balancing act
Probably one of the hardest challenges that I am facing with writing the book is to know what to say and what to leave unsaid. Phrasing it another way, it is choosing at what level to talk to the reader. On the one hand, I really want the reader to be able to make immediate use of the concepts that I am talking about, which drive me to do more practical demonstrations, code samples and covering more common situations. On the other hand, those take up a lot of room, and they tend to be boring...
Is this valuable?
Going back to the theme of visually editing a DSL, we have something like this: The backend to this cutey is pattern recognition with custom UI on top. You have to teach it about each pattern that you use with it, but you can get pretty smart with how it works while using a fairly brute force (and simple) techniques. Thoughts?
Nullifying Null
One of the more annoying problems with building rules that are also code is that you have to deal with code related issues. One of the more common ones is NullReferenceException. For example, let us say that we have the following rule: when Order.Amount > 10 and Customer.IsPreferred:
ApplyDiscount 5.precent
We also support a mode in which a customer can create an order without actually registering on the site (anonymous checkout).
In this scenario, the Customer property is null. We can rewrite the rule to look like this:...
How to execute a set of statements in an expression
One of the most common problems with using Boo's Meta Methods is that they can only return expressions. This is not good if what you want to return from the meta method is a set of statements to be executed. The most common reason for that is to initialize a set of variables. Obviously, you can call a method that will do it for you, but there is a simpler way. Method invocation is an expression, and anonymous delegate definition is also an expression. What does this tells you? That this is an expression as well: ...
Inline Anonymous Visitors
One of the most common chores when working with compilers is the need to create special visitors. I mean, I just need to get a list of all the variables in the code, but I need to create a visitor class and execute it in order to get the information out. This is not hard, the code is something like this: public class ReferenceVisitor : DepthFirstVisitor
{
public List<string> References = new List<string>();
public void OnReferenceExpression(ReferenceExpression re)
{
...
Thinking about chapter 12 - DSL Patterns
I am giving a lot of thought to this chapter, because I want to be able to throw out as much best & worst practices as I can to the reader. Here is what I have right now: Auditable DSL - Dealing with the large scale - what the hell is going on? User extensible languages Multi lingual languages Multi file languages Data as a first class concept Code == Data == Code ...
DSL Dialects
Let us take this fancy DSL: And let us say that we want to give the user some sort of UI that shows how this DSL works. The implementation of this DSL isn't really friendly for the UI. It was built for execution, not for display. So how are we going to solve the problem? There are a couple of ways of doing that, but the easiest solution that I know of consists of creating a new language implementation that is focused on providing an easy to build UI....
Code Data Mining
I just wrote this piece of code: class ExpressionInserterVisitor : DepthFirstVisitor
{
public override bool Visit(Node node)
{
using(var con = new SqlConnection("data source=localhost;Initial Catalog=Test;Trusted_Connection=yes"))
using (var command = con.CreateCommand())
{
con.Open();
command.CommandText = "INSERT INTO Expressions (Expression) VALUES(@P1)";
...
Didja know? #Develop has a class designer
This is a sample: I am finding #Develop to be a treasure throve of components that can be used for DSL development... with the added bonus of being accessible and easy to use.
Primitive Contextual Intellisense
The last time we looked at this issue, we built all the pieces that were required, except for the most important one, actually handling the contextual menu. I am going to be as open as possible, Intellisense is not a trivial task. Nevertheless, we can get pretty good results without investing too much time if we want to. As a reminder, here is the method that is actually responsible for the magic that is about to happen: public ICompletionData[] GenerateCompletionData(string fileName, TextArea textArea, char charTyped)
{
return new ICompletionData[] {
...
Code or data?
Here is a question that came up in the book's forums: I can't figure out how to get the text of the expression from expression.ToCodeString() or better yet, the actual text from the .boo file. It appears to automagically convert from type Expression to a delegate. What I want is to be able to when a condition is evaluated display the condition that was evaluated, so if when 1 < 5 was evaluated I would be able to get the string "when 1 < 5" - Any way to do this? ...
Strategies for editing DSL scripts in production
Another interesting question from Chris Ortman: So I write my dsl, and tell my customer to here edit this text file? How do I tell them what the possible options are? Intellisense? This is a web app, and my desire to build intellisense into a javascript rich text editor is very low. It might be a good excuse to try out silverlight but even then it seems a large task. Or I put...
Basic intellisense
In many cases, intellisense is the killer feature that will make all the difference in using a language. However, it is significantly harder than just defining the syntax rules. The main problem is that we need to deal with the current context. Let us take a look at what we would like our intellisense to do for the Quote Generation DSL. On empty line, show "specification" On specification parameter, show all available modules. On empty line inside specification block, show all actions (requires, users_per_machine, same_machine_as) ...
Custom Syntax Highlighting
Just using the Boo syntax isn't really enough in many cases, you want to handle your own custom keywords, behaviors, etc. #Develop make this a piece of cake, since it defines the syntax highlighting using an XML file, and handles the actual parsing and coloring on its on. Here is the overall structure of such a file: <?xml version="1.0"?>
<SyntaxDefinition name="Boo"
extensions=".boo">
<Environment>
<Default bold="false"
...
Smarter Syntax Highlighting
My previous approach isn't really suited to anything but the most trivial of tasks. What we can do, however, is to utilize existing components that already built this functionality. Let us say that we want to embed a Boo's code editor in our application. Here is what we need to do: public class TextEditorForm : Form
{
public TextEditorForm()
{
var editorControl = new TextEditorControl
{
...
Poor man's syntax highlighting
Before getting to the more complex scenarios of creating professional DSL, I wanted to start by showing how you can easily create your own syntax highlighting. This was the result: And this is the actual code that makes this happens: private void codeTextBox_TextChanged(object sender, EventArgs e)
{
int prevSelectionStart = codeTextBox.SelectionStart;
int prevSelectionLength = codeTextBox.SelectionLength;
codeTextBox.SelectionStart = 0;
codeTextBox.SelectionLength = codeTextBox.TextLength;
codeTextBox.SelectionColor = DefaultForeColor;
var keyWords = new[] { "specification", "requires", "users_per_machine", "same_machine_as" };
...
Help: Can you think about a UI representation?
I am currently writing the chapter about creating professional DSL. One of the subjects that I am dealing with is creating graphical representations of a textual DSL. Here is the DSL script in question: specification @vacations:
requires @scheduling_work
requires @external_connections
specification @salary:
users_per_machine 150
specification @taxes:
users_per_machine 50
specification @pension:
same_machine_as @health_insurance
I am not sure what kind of UI representation this should have.
For that matter, let us take the example that I commonly use, of a rule engine for an ordering system:
when order.total > 1000 and order.payment_method = 'CreditCard':
apply_discout_of 5.precent
How would you visualize that?
Nicer Linq
A few days ago I posted about Ugly Linq. Ever since then, I kept thinking about how ugly it is to handle this by hand. Suddenly, it hit me that I don't have to do it that way. Boo already has the facilities to take a compiler AST and translate that into the code that would recreate this AST. In particular, this makes the code we previously had to write to this: public class ConditionMacro : AbstractAstMacro
{
public override Statement Expand(MacroStatement macro)
{
Expression serialize = new CodeSerializer().Serialize(macro.Arguments[0]);
var body = new Block();
body.Statements.Add(new ReturnStatement(macro.Arguments[0]));
return new ExpressionStatement(
new MethodInvocationExpression(
AstUtil.CreateReferenceExpression(typeof(Condition).FullName),
new BlockExpression(body),
serialize
)
);
}
}
And what...
DSL Patterns - Multi File DSLs
One of the common misconceptions about a DSL is that it to think about each DSL script independently. As a simple example, let us go back to the routing DSL, we may have a rule like this: priority 10when msg is NewOrder and msg.Amount > 10000: dispatch_to "orders@strategic"
And our focus is mainly on the actual language and syntax that we want to get. This is a mistake, because we aren't considering the environment in which the DSL lives. In the same sense that we rarely consider a code file independently, we should...
DSL Patterns - User Extensible Language
One of the common mistakes in a DSL is to create one that solve a demo problem and leave it at that. As a simple example, let us say that we build a DSL to handle order management. Here is a typical scenario that was talked about during development: when is_preferred_user and order_amount > 500:
apply_discount 5.precent
Sounds reasonable, right?
Except, that this isn't really a good way to express a business rule. A common business rule usually have a lot more complexity to it. Here is a good example:
when user.payment_method is credit_card and ((order_amount >...
Multi Tenancy - Scriptability and DSL
Previously on the Multi Tenancy series:
Multi Tenancy - Definition and scenario.
Reviewing Litware HR - P&P Multi Tenancy Sample.
Multi Tenancy - The Physical Data Model
Multi Tenancy - Extensible Data Model
Multi Tenancy - Extensible Behaviors
Multi Tenancy - Development Structure
Multi Tenancy - Where do you put variability?
It probably would not surprise you that I a seeing a place for a DSL in a Multi Tenant application. Indeed, one might almost say that I have DSL on my mind a lot these days.
Multi Tenant applications are all about managing variability...
Solution: What is the problem?
I asked what is wrong with the following DSL: specification @vacations: requires @scheduling_work requires @external_connections I also constrained the problem so it can't be any of the following: It has nothing to do with the implementation. It has nothing to do with the actual engine running this. Look for what isn't there. It is only a problem when you scale up. ...
Writing Domain Specific Languages in Boo: Chapter 10 - Creating professional DSL - Table of Contents
An incomplete list of subjects for the 10th chapter.
The editor environment
Color highlighting
Intellisense
Sharp Develop
Visual Studio
DSL Code Generation
The Code DOM Provider for Boo
Specific DSL Writers
Graphical UI
Displaying the DSL
Saving DSL modifications
The value of a graphical front end
Errors and warnings handling
Debugging
Dealing with the large scale - what the hell is going on?
Building Domain Specific Languages with Boo: Chapter 9 - Versioning
At long last, I am done with Chapter 9. The actual writing process didn't took so long, but the ideas took a long time to come into a coherent form. Here is the final table of content: 9.1 Starting from a stable origin 9.2 Planning our DSL's versioning story 9.2.1 Implications of modifying the DSL engine 9.2.2 Implications of modifying the DSL API and Model 9.2.3 Implications of modifying the DSL Syntax ...
Challenge: What is the problem?
This code is part of a DSL that is used to generate quotes. In particular, this bit is used to build the dependency graph for the engine to run on. specification @vacations:
requires @scheduling_work
requires @external_connections
There is a serious design issue with this bit of code. Can you figure it out?
Hints:
It has nothing to do with the implementation.
It has nothing to do with the actual engine running this.
Look for what isn't there.
It is only a problem when you scale up.
And no, there are no performance problems whatsoever.
The Clippy Compiler
I am working on the versioning chapter for the book, and I am current at the place where I am suggesting letting the compiler know about what kind of situations your users are likely to get into. The examples I am using is changing the API from: requires @vacations To: requires @vacations, "Some explanatory text" And handling the scenario where the user attempts to write to the old API. Here is the default approach: BCE0017: Boo.Lang.Compiler.CompilerError: The best overload...
Building Domain Specific Languages in Boo: Chapter 9 TOC
Thoughts? Starting from a stable origin Planning our DSL's versioning story Implications of modifying the DSL Engine Implications of modifying the DSL API and Model Implications of modifying the DSL Syntax Implications of modifying the DSL Environment Regression Test Suite Versioning cut off point - where to apply versioning concerns Versioning strategies: ...
A DSL is also its usage
The title says it all, I think. Okay, it doesn't, I admit. There is a lot of focus with DSL about the syntax. And there is some focus on the engine. There is very little focus about how both the environment and the usage of the DSL affect the DSL itself. Here are a few examples: Naming convention Script ordering Execution location (when you are running the scripts) All of those are of particular importance in many DSL, not only for the actual execution, but for how...
Notes on versioning Domain Specific Languages
Those are just a few topics that I feel are important for discussion when talking about versioning DSL: Different behavior at runtime API vs. Syntax Different dialects Backward and forward Compatibility Pros: Keeping existing assets Training Knowledge The Test of Fire Cons: Increased costs ...
Boo Migration DSL
Nathan Stott is doing some really interesting things with Rhino DSL and Boo. His latest post outlines how to create this syntax: CreateTable "Cats":
Int32 "Id", { "identity" : true, "primary" : true }
String "Name", { "length" : 50 }
I like it.
Simple State Machine
Nathan has posted Simple State Machine to CodePlex, it is the first project that I am aware of that uses Rhino DSL and the techniques that I am talking about in the book. What is impressive about this is the level of professionalism that is involved in the project. It is a full scale DSL, with all the supporting infrastructure. I spent half an hour or so going through the entire thing, and I am impressed. Put simply, this is how I think state based work flows should be defined. I could easily see myself extending this a...
Taking conventions to their obvious conclusion: The mandatory test language
I am considering having a language that mandates tests. If you don't have a matching test for the code in question, it will refuse to run. If the tests fail, it will refuse to run. If the tests takes too long, they are considered failed and the code will refuse to run. This certainly ensure that there would be test. It wouldn't ensure that they would be meaningful, however. That is fine by me. I am not interested in policy through enforcement, just gentle encouragement in the right direction. The technical challenges of implementing such a system...
The magic of boo - Flexible syntax
when I am writing DSL, I keep hitting one pain point. The CLR naming conventions, which are more or less imprinted on my eyelids, are not really conductive to clear reading in a DSL. Let us take these entities, and see what we get when we try to build a DSL from them: The DSL is for defining business rules, and it looks like this: when User.IsPreferred and Order.TotalCost > 1000:
AddDiscountPrecentage 5
ApplyFreeShipping
when not User.IsPreferred and Order.TotalCost > 1000:
SuggestUpgradeToPreferred
ApplyFreeShipping
when User.IsNotPreferred and Order.TotalCost > 500:
ApplyFreeShipping
The main problem with this style of...
Testing Domain Specific Languages
Roughly speaking, a DSL is composed of the following parts: It should come as no surprise that when we test it, we test each of those components individually. When the time comes to test a DSL, I have the following tests: CanCompile - This is the most trivial test, it assert that I can take a known script and compile it. Syntax tests - Didn’t we just test that when we wrote the CanCompile() test? When I am talking about testing the syntax I am not talking about just...
Testing DSL Syntax with Interaction Based Testing
How do I test the syntax in this DSL? HandleWith should translate to a method call with typeof(RoutingTestHandler) and a delegate. import BDSLiB.Tests
HandleWith RoutingTestHandler:
lines = []
return NewOrderMessage( 15, "NewOrder", lines.ToArray(OrderLine) )
Well, I use interaction based testing, obviously. I find this test utterly fascinating, because it is fairly advance, in a roundabout sort of way, and yet it is so simple.
[Test]
public void WillCallHandlesWithWithRouteTestHanlderWhenRouteCalled()
{
const IQuackFu msg = null;
var mocks = new MockRepository();
var routing = dslFactory.Create<RoutingBase>(@"Routing\simple.boo");
var mockedRouting = (RoutingBase)mocks.PartialMock(routing.GetType());
Expect.Call(() => mockedRouting.HandleWith(null, null))
.Constraints(Is.Equal(typeof(RoutingTestHandler)), Is.Anything());
mocks.ReplayAll();
mockedRouting.Initialize(msg);
mockedRouting.Route();
mocks.VerifyAll();
}
Easy extensibility: xUnit integration for DSL
I saw several solution for extending NUnit and MbUnit to add new functionality, all of them were far too complex for me. I didn't want this complexity. Here is the entire code that I had to write in order to make xUnit integrate with my DSL: public class DslFactAttribute : FactAttribute
{
private readonly string path;
public DslFactAttribute(string path)
{
this.path = path;
}
protected override IEnumerable<ITestCommand> EnumerateTestCommands(MethodInfo method)
{
DslFactory dslFactory = new DslFactory();
dslFactory.Register<TestQuoteGeneratorBase>(
new TestQuoteGenerationDslEngine());
TestQuoteGeneratorBase[] tests = dslFactory.CreateAll<TestQuoteGeneratorBase>(path);
for(var test in tests)
{
Type dslType = test.GetType();
BindingFlags flags = BindingFlags.DeclaredOnly |
BindingFlags.Public |
BindingFlags.Instance;
foreach (MethodInfo info in dslType
.GetMethods(flags))
{
if (info.Name.StartsWith("with"))
yield return new DslRunnerTestCommand(dslType, info);
}
}
}
}
And the DslTestRunnerCommand:
public class DslRunnerTestCommand...
Integrating a test DSL with unit testing framework: Success
Some of the things that I come up when I am doing this things are simply hilarious. It start from this: And moves to this: I don't think that anyone can say that the naming of those unit tests are in any way misleading.
Create a test DSL to test the DSL
Yesterday I asked how we can efficiently test this piece of code: specification @vacations:
requires @scheduling_work
requires @external_connections
Trying to test that with C# code resulted in 1500% disparity in number of lines of code. Obviously a different approach was needed. Since I am in a DSL state of mind, I wrote a test DSL for this:
script "quotes/simple.boo"
with @vacations:
should_require @scheduling_work
should_require @external_connections
with @scheduling_work:
should_have_no_requirements
I like this.
You can take a look at the code here.
Unit testing a DSL
There is something that really bothers me when I want to test this code: specification @vacations:
requires @scheduling_work
requires @external_connections
And I come up with this test:
[TestFixture]
public class QuoteGenerationTest
{
private DslFactory dslFactory;
[SetUp]
public void SetUp()
{
dslFactory = new DslFactory();
dslFactory.Register<QuoteGeneratorRule>(new QuoteGenerationDslEngine());
}
[Test]
public void CanCompile()
{
QuoteGeneratorRule rule = dslFactory.Create<QuoteGeneratorRule>(
@"Quotes/simple.boo",
new RequirementsInformation(200, "vacations"));
Assert.IsNotNull(rule);
}
[Test]
public void WhenUsingVacations_SchedulingWork_And_ExternalConnections_AreRequired()
{
QuoteGeneratorRule rule = dslFactory.Create<QuoteGeneratorRule>(
@"Quotes/simple.boo",
new RequirementsInformation(200, "vacations"));
rule.Evaluate();
SystemModule module = rule.Modules[0];
Assert.AreEqual("vacations", module.Name);
Assert.AreEqual(2, module.Requirements.Count);
Assert.AreEqual("scheduling_work", module.Requirements[0]);
Assert.AreEqual("external_connections", module.Requirements[1]);
}
[Test]
public void WhenUsingSchedulingWork_HasNoRequirements()
{
QuoteGeneratorRule rule = dslFactory.Create<QuoteGeneratorRule>(
@"Quotes/simple.boo",
new RequirementsInformation(200, "scheduling_work"));
rule.Evaluate();
Assert.AreEqual(0, rule.Modules.Count);
}
}
I mean, I heard about disparity in number of lines, but I think that this is beyond ridiculous.
It is not the parser I fear
Martin Fowler talks about the almost instinctive rejection of external DSLs because writing parsers is hard. I agree with Fowler on that writing a parser to deal with a fairly simple grammar is not a big task, certainly there isn't anything to recommend XML for the task over an textual parser. The problem that I have with external DSL is actually different. It is not the actual parsing that I object to, it is the processing that needs to be done on the parse tree (or the XML DOM) in order to get to an interesting result that I...
Compare and contrast: Rhino Mocks vs. Hand Rolled Stubs
For various reasons which will be made clear soon, I needed to write the same test twice, once using Rhino Mocks, the second time using hand rolled stubs. I thought it was an interesting exercise, since this is not demo level code. [Test]
public void WillCallHandlesWithWithRouteTestHanlderWhenRouteCalled_UsingRhinoMocks()
{
const IQuackFu msg = null;
var mocks = new MockRepository();
var routing = dslFactory.Create<RoutingBase>(@"Routing\simple.boo");
var mockedRouting = (RoutingBase)mocks.PartialMock(routing.GetType());
Expect.Call(() => mockedRouting.HandleWith(null, null))
.Constraints(Is.Equal(typeof(RoutingTestHandler)), Is.Anything());
mocks.ReplayAll();
mockedRouting.Initialize(msg);
mockedRouting.Route();
mocks.VerifyAll();
}
[Test]
public void WillCallHandlesWithWithRouteTestHanlderWhenRouteCalled()
{
const IQuackFu msg = null;
dslFactory.Register<StubbedRoutingBase>(new StubbedRoutingDslEngine());
var routing = dslFactory.Create<StubbedRoutingBase>(@"Routing\simple.boo");
routing.Initialize(msg);
routing.Route();
Assert.AreEqual(typeof (RoutingTestHandler), routing.HandlerType);
Assert.IsInstanceOfType(
typeof(NewOrderMessage),
routing.Transformer()
);
}
Open seats for the DSL course
There are still open seats for the DSL course in Austin, next week (19th - 20th May).
You can register here: http://ayende.eventbrite.com/
Course: Building Domain Specific Languages in Boo
You can register here for a two days course in building DSL with Boo.
It is going to take place two weeks from today, in Austin. (19 - 20 May)
I know that this is short notice, but it wasn't something that was planned well in advance. It came out of the ALT.Net conference.
Topics:
Creating Domain Specific Languages
The Boo Language
Flexible compiler and malleable language
Creating applications with embedded DSL
Management, tracing and debugging
Tooling support
Testing and maintainability concerns
There are ten seats open for that.
I hope we would have fun.
I would also like...
DSL Article on InfoQ
My DSL article has been published on InfoQ!
You can get it here
The quote generation DSL
I am doing some work on the DSL book right now, and I run into this example, which is simple too delicious not to post about. Assume that you have the following UI, which you use to let a salesperson generate a quote for your system. This is much more than just a UI issue, to be clear. You have fully fledged logic system here. Calculating the total cost is the easy part, first you have to understand what you need. Let us define a set of rules for the application, is will...
New version of the book out!
Chapters 5 - 7 are now available, covering a lot of the building of a DSL and integrating that into an application. This version contains a lot of fixes from reviewers, but I intend to make some chapter shuffling which will put the hard core stuff toward the middle of the book, after we cover more interesting scenarios. Building Domain Specific Languages with BOO
Building Domain Specific Languages in Boo: Available for early access
Well, this is a big moment. My book is now available for as part of the Manning Early Access Program. Chapter 1 to 4 are already there, but they are there in their nearly raw format. That means that they have not been edited yet, and I didn't put in the results from the review cycle that we did yet. The book forum is here, I eagerly awaits the posts telling me how off base I am :-) More seriously, I would appreciate any feedback that you have on the book, the topic, the code, etc. Thanks, ~Ayende
Fun with DSL - Didja know the CLR allows it?
I swear that I didn't mean to do it, but it just happened.
Refactoring for Separation Of Concerns: A real world example
I just did a major refactoring to Rhino DSL, to add orthongality to the DslEngine. It was an interesting refactoring, and something that is worth talking about. Let us take a look at the before image, first: We have two classes that are involved in creating and managing DSL. Notice the FileFormat, CreateInput, GetMatchingUrlsIn and NotifyOnChange methods on the DslEngine? Or the GetFromCache, RemoveFromCache, SetInCache methods? The firsts are related to the storage of the DSL scripts on disk and the second batch are related to caching, they have nothing to do with the actual work performed by...
Debugging DSL
Well, that is fairly easy, all you need it to press F11 :-)
Purely declarative DSL
Let us assume that we need to build a quote generation program. This mean that we need to generate a quote out of the customer desires and the system requirements. The customer's desires can be expressed in this UI: The system requirements are: The Salary module is specification is a machine per every 150 users. The Taxes module requires a machine per 50 users. The Vacations module requires the Scheduling Work module. The Vacations module requires the External Connections module. The Pension Plans module requires the External Connections module. The Pension Plans module must be on...
Authorization DSL
Here is a tidbit that I worked on yesterday for the DSL book: operation "/account/login"
if Principal.IsInRole("Administrators"):
Allow("Administrators can always log in")
return
if date.Now.Hour < 9 or date.Now.Hour > 17:
Deny("Cannot log in outside of business hours, 09:00 - 17:00")
And another one:
if Principal.IsInRole("Managers"):
Allow("Managers can always approve orders")
return
if Entity.TotalCost >= 10_000:
Deny("Only managers can approve orders of more than 10,000")
Allow("All users can approve orders less than 10,000")
There is no relation to Rhino Security, just to be clear.
I simply wanted a sample for a DSL, and this seems natural enough.
The applicability of DSL
When is a DSL applicable? When will using a DSL make our job easier? There are several layers to the answer, depending on the type of DSL that we want, and the context that we want to apply it. When building a technical DSL, we will generally use that as bootstrapping and configuration mechanism, to make it easier to modify and change a part of the system. In general, those DSL are focused enabling recurring types of tasks, usually of one off nature. Configuration is a common scenario, since it is probably the simplest to explain and to start, but...
Looking for a DSL idea
I need to start writing the second part of the book soon. This one is supposed to take a DSL implementation through all the interesting stages that I would like to cover. However, I am not sure yet what the subject of the DSL will be. I need something that has enough scope to last for about a hundred pages, complex enough to expose usual problem when writing DSL and not tied to a specific domain so strongly that it would be hard to outsiders to grasp. I am also interested in knowing what kinds of patterns and problems...
Deploying from source control
This is just a quick note for public review, you are probably aware that I am doing deployments by doing a "svn up && build". I am now thinking about how we can apply the same idea to deploying DSL. This ensures, at the very least, that our DSL are under source control. But that has led me to another thought, if we are enforcing SCM for the DSL, why not enforce unit testing as well? Part of the loading process of a DSL can be loading the DSL and its unit tests, executing the unit tests and only accepting...
Throwing a DSL over the wall
Here are a few thoughts about the idea that given a DSL, you can just turn your entire business related stuff to the business users, and be done with it. This kind of thinking is very common, from the dreams of the secretary building applicaitons by dragging boxes to the business analyst modifying the workflows. I disagree with this quite strongly. Giving a user this type of tool is an irresponsible step at best. The users are not developers, and they will make mistakes. And the first time that the business will lose money over it, it will be your fault....
Requirements of a DSL engine
Getting a DSL to work for a single file is fairly easy. Getting it to work in a real application is much harder. There are quite a few scenarios that you need to deal with irrespective of the DSL itself. Those issues include: Caching - compilation is expensive Batch compilation - see above, also very important Error handling - we can probably do at least some work here to make it easier. At the very least, output something like compilation exception would give us the nice yellow page with the error, compiled output, etc. Automatic refresh - when...
DSL & IoC - Combining powerful concepts
I was just writing this line of code: DslExecuter<SchedulingDslEngine>.Create(“validateWebSiteIsUp”)
When it occurred to me that this looks a look like an IoC resolve call. Which got me thinking about combining both concepts together, which gave me the shakes.
The idea is that we can expose those dependencies like this:
customers as ICustomerRepository
fraudService as IFraudService
alerts as IAlertService
for customer in customers.FindAll(With.Orders):
for order in customer.Orders:
if fraudService.IsFraud(order):
alerts.ForFaurdOn(order)
if customer.Important:
alerts.NotifyCustomerAboutFraud(customer)
Leaving aside the mountain slope code*, consider what is going on here. We are exposing dependencies by specifying the types that we need. We can then get them through the IoC and execute the code in hand.
This means that the...
Building an External DSL
Over in the Alt.net mailing list, Glenn said: If I was really being verbose in the way a business user would expect to define this I would say When customer is preferred and the order exceeds 1000 then apply a .05 discount and apply free shipping. Now do that in a DSL…. Now, I am sucker for this kind of things, and I can probably use the practice. So I decided to go ahead with this. Just to be clear, this is supposedly a natural language processing, only we are going to cheat and get around that....
DSL Styles - Imperative vs. Declarative
I posted yesterday about building a rule engine using a DSL. After I wrote that, I tried to think about what style it was. That made me realize that I had no clear distinction between imperative and declarative DSL. At least not a coherent, internally consistent one. I asked the Alt.net mailing list, and a lively discussion enthused. Contrary to my opinion, a lot of people disagreed with my opinion that this is a declarative style DSL: when User.IsPreferred and Order.TotalCost > 1000: addDiscountPrecentage 5 applyFreeShippingwhen not User.IsPreferred and Order.TotalCost > 1000:suggestUpgradeToPreferred applyFreeShippingwhen User.IsNotPreferred and Order.TotalCost >...
A rule engine in less than 70 lines of code
As I continue to explore what we can do with DSL, I am getting more and more excited. Let us take a look at the following syntax: when User.IsPreferred and Order.TotalCost > 1000:
addDiscountPrecentage 5
applyFreeShipping
when not User.IsPreferred and Order.TotalCost > 1000:
suggestUpgradeToPreferred
applyFreeShipping
when User.IsNotPreferred and Order.TotalCost > 500:
applyFreeShipping
The backend for that is a simple 68 lines class. Again, we had to extend the language to support the when keyword, but that is all we really had to do.
I tried...
A time for a DSL
Most of the time, I use a DSL as a way to setup and configure some sort of a runtime engine. We will get to that later when we will talk about imperative vs. declarative DSL. But the rule of the thumb that I use is that I’ll use a Fluent Interface if I usually need this functionality at the same time I am developing. If I need to be expressive at a different time than the development of the application, then it probably means that I should use a DSL instead. The issues of complexity vs. expressiveness also come...
If it is not dynamic vs. static - what is it?
I am looking for a term to describe languages like C# or Java vs. languages like Boo and Ruby. At first I thought about using the usual static and dynamic descriptions, until I remembered that Boo is a static language. I am thinking about rigid vs. flexible languages, in terms of syntax and expressiveness. Any comments?
Deconstructing DSL Evolution
You could say that I have an interest in DSL at the moment, which is why I was excited when I saw this post. It talks about how you can create natural English as a BDD specification. In fact, the final "code" in the post looks like this: Scenario: savings account is in credit
Given my savings account balance is 100
And my cash account balance is 10
When I transfer 20
Then my savings account balance should be 80
And my cash...
Implementing a DSL
Tim Wilde asked how I would build a DSL, given the example that I gave: task "warn if website is not alive":
every 3.Minutes()
starting now
when WebSite("http://example.org").IsAlive == false
then:
notify "admin@example.org", "server down!"
Now, I have a small confession to make, I didn't build the DSL for the post, I just thought it up and posted it. Tim has spiked a somewhat nicer implementation of the C# fluent interface, and it was about 250 lines of code, in 8 classes, 6 interfaces.
Face with that, I felt that I had no other choice than to go on and build a DSL implementation. If fact, I think...
Why C# doesn't have extension properties
I just run into this problem, and I came up with a different reason than the usual one. C# simply doesn't have the concept of indexed properties. This is not legal C# code: public static string Items[string something]
{
get { return something; }
}
I think you can do that with VB.Net, and I am certain that C++ supports it.
The interesting part is that I run into it while building a DSL. The limit of the implementation language has actually limited the DSL itself.
DSL vs. Fluent interface: Compare & Contrast
Let us take a simple example, scheduling tasks, and see how we can demonstrate the difference between the two. From the perspective of the client, what do we want from a scheduling tasks? Define named tasks Define what happens when the task is executed Define when a task should execute Define conditions to execute the task Define recurrence pattern Now, let us look at the fluent interface for this: new FluentTask("warn if website is not alive")
.Every( TimeSpan.FromMinutes(3) )
.StartingFrom( DateTime.Now )
.When(delegate
{
return WebSite("http://example.org").IsAlive == false;
})
.Execute(delegate
{
Notify(“admin@example.org”, “server down!”);
});
And contrast that with the DSL sample:
task "warn if website...
Why we need for Domain Specific Languages
I speak quite often about DSL and how to build them, but so far I did not do was explain why you need a DSL at all. After all, since you are reading this blog, you already know how to program. Can’t we just use “normal” programming languages to do the same job? We can even do with a dash of fluent interfaces and Domain Driven Design to make the code easier to read. We need to inspect the different needs that lead the drive toward a DSL. A Technical DSL A technical DSL is supposed to be consumed...
How to visualize a Domain Specific Language
Andrey Shchekin made a good point when he commented on my post about graphical DSL: I do not use DSLs that are purely graphical. But talking about documentation, I think it is very useful to have a graphical _representation_ for some DSLs that is synchronized with (or generated from) the DSL itself.For example, any workflows or dataflows (think Rhino ETL) are much easier to see at a glance on visual surface. Editing them on visual surface is also an option, but not a requirement. I certainly am a believer that a DSL should be mostly declarative, and indeed, taking Rhino...
Domain Specific Language: Losing the original language
Here is an interesting question, at what point you drive away so far from the original language that you lose the benefits of an internal DSL? The following examples are several ways to represent the same thing, going from the extreme DSL to the no DSL to C#. I think that they all have value, and neither cross that line, but I believe that they will help make the point. The example here is a a simple MonoRail Controller: #1: controller Field: action Tag: @defaultTags = [ "green", "trees", "growth" ] #2: controller Field: def Tag(): @defaultTags = [ "green",...
Binsor 2.0
All credit should go to Craig Neuwirt, for some amazing feats of syntax. He has managed to extend Binsor to include practically all of Windsor's configuration schema. This is important because previously we had to resort to either manual / ugly stuff or go back to XML (see: manual & ugly). This is important because some of the more interesting things that you can do with Windsor are done using the facilities, and Craig has made sure that Binsor will support the main ones in a natural manner, including out of the box support for the standard configuration model. Let...
Analyzing a DSL implementation
You were just handed a strange DSL implementation, it does stuff, and it may be cool, but you have no idea how it works. Craig Neuwirt recently did a major overhaul of Windsor ( I am going to post soon with the details of how cool it is now ), but I am now faced with the question, how do you grok such a thing? I thought that it would be useful to put a list of what I am doing to understand how the internals now works. It goes without saying, but the nitpickers will ask, that a...