Ayende @ Rahien

DotNetRocks on Domain Specific Languages

Tue, 02 Mar 2010 18:59:00 GMT

Last week I recorded a DNR session about Domain Specific Languages.

I also talked just a bit about writing your own DSL, some challenges that I run into since the book, the Boo language and why it is suitable for DSLs and how the entire process works.

Looking forward for your comments.

DSLs in Boo discount offer

Tue, 02 Feb 2010 18:04:00 GMT

Manning is running a special today for my book, 50% off print & ebook versions.

Just use the following code: boo50

Note that this code is valid only for today, so hurry up!

DSLs in Boo is out!

Sat, 23 Jan 2010 10:00:00 GMT

It has been quite a journey for me, starting in 2007(!) up until about a month ago, when the final revision is out. I am very happy to announce that my book is now available in its final form.

When I actually got the book in my hands I was ecstatic. That represent about two years worth of work, and some pretty tough hurdles to cross (think about the challenge that editing something the size of a book from my English is). And getting the content right was even harder.

On the one hand, I wanted to write something that is actionable, my success criteria for the book is that after reading it, you can go ahead and write production worthy Domain Specific Languages implementations. On the other hand, I didn’t want to have the reader left without the theoretical foundation that is required to understand what is actually going on.

Looking back at this, I think that I managed to get that done well enough. The total page count is ~350 pages, and without the index & appendixes, it is just about 300 pages. Which, I hope, is big enough to give you working knowledge without bogging you down with too much theory.

My book is on Manning – deal of the day

Tue, 15 Dec 2009 06:45:00 GMT

My book is on Manning’s Deal of the Day. DSLs in Boo: Domain Specific Languages in .NET – get the ebook for $10 by using this link.

The journey for the book is done, so it is the full deal and the final version that you’ll be getting.

Help needed: Writing Domain Specific Languages in Boo – Java Edition

Thu, 23 Apr 2009 16:10:49 GMT

Just came out of a discussion with Manning about my book (which is approach the last few stages before actually printing!), apparently they hit upon the notion that Boo works on both the CLR and the JVM, and are interested in having a Java edition of the book.

Disclaimer: This is early, and anything is subject to change, blah blah blah.

I find this hilarious, since this is Hibernate in Action vs. NHibernate in Action, in reverse. At least, I hope it is. The problem? I am not familiar enough with Java to be able to write a book targeting it, hence this post.

If you are familiar with Java and BooJay (filter 90%), read my book (filter 90%) and think you could help (filter 0%), I would love to talk to you.

M is to DSL as Drag & Drop is to programming

Mon, 23 Mar 2009 12:21:22 GMT

I have been sitting on this post for a while now, because that was my first impression after seeing Oslo & M and all the hype around it from the PDC. To be frank, I had a hard time believing my own gut feeling. I kept having the feeling that I am missing something, which is why I avoided talking about this so far.

But, as time passed, and as we started to see more and more about Oslo and M, it validated my initial thinking. Now, just to be clear, I don’t intend to even touch on the whole of Oslo in this post. I don’t have a problem stating that I still don’t see the whole point there, but that is beside the point (no pun intended). What I would like to talk about is the M language, its usage, and the DSL that Microsoft shows as samples.

I see M as a whole lot of effort trying to optimize something that is really not that interesting, complex or really very hard. I look at the M language, the way that you worked with, the tooling and the API and I would fully agree that it is a nice parser generator.

What it is not, I have to say, is a DSL toolkit. It is just one, small, part of building a DSL. And, to be perfectly honest, M is the drag & drop of DSL. It looks good, on first glance, but then you dig just a little deeper and you see what actually going on, and you realize that you are probably not where you wanted to be.I see it as trying very hard to optimize opening the car’s door. While I assume that this is interesting to someone in the world, optimizing the opening of a car door is crucial, I don’t really see it as an important feature. More to the point, it has negligible effect on the time taken for the primary task for which we use a car, the actual driving!

Why am I saying that?

Well, M is used for defining the syntax of the language, which is what most people look at. It does a good job there, but it also stops there. And there is a lot of stuff other then the syntax that you really care about.

Here is a snippet from MisBehave, which was an attempt to build a BDD framework on top of M:

Pretty impressive syntax, right?

The problem is that there isn’t really a good way to take this and translate that into something that is executable. Not without doing a lot of work. And that is why I am saying that M isn’t really an important piece of the stack. The actual syntax definition isn’t really that important. It is all the other things that you do with the DSL that matters.

Let us take a look at MUrl:

I am looking at this, and after looking at the source code, I still can’t figure out the point.

Yes, this is a demo DSL. But it is a good example that shows how you can completely miss the point with regards to a DSL. What problem does this DSL solve? What benefits do I get from integrating that into my system?

How does this helps me solve a real problem?

The answer is that it doesn’t. The only remotely useful case that I can think of is if I really want to be able to issue REST calls from the command line, and even then, there are better ways of doing that on the command line. MUrl is an exercise in abstraction for the sake of abstraction. More than that, it gives the impression that it is something that is is not.

If you want to show me a DSL, show me one that has logic, not one that is a glorified serialization format. That is the sweet spot for a DSL, to extract policy decisions from your systems, so you can work with them at a higher level and have easier time making change.

M is not a language for creating DSL. It is a language to define a serialization format, that is all.

Building Domain Specific Languages with Boo – Full book now available

Mon, 02 Mar 2009 00:22:20 GMT

Building Domain Specific Languages with Boo

Another big milestone, early access subscribers can now read the entire book, all 13 chapters and both appendixes.

Not edited yet, but much closer to completion.

DSL: Tests as documentation

Mon, 09 Feb 2009 10:48:11 GMT

I have several DSL that have no documentation beyond their source and the tests. They are usable, useful and have been of a lot of help. However, I have run into situations where I, as the language author, could not answer a question about the language without referring to the code. I strongly recommend in investing the time to create good documentation in your DSL.

Even if you are using a Behavior Driven Design flavored tests, it is not quite enough. Those types of tests can help make it clear what the language is doing, but they are not the type of documentation that you can hand to an end user and expect them to start using the language.

Even if your users are developers, it is not nearly good enough approach. It is your responsibility to make the system easy to use for the users, and documentation is a key part of that.

Handing them the tests is a good way to handle the complex cases, if your users are developers, but it is not a good way to reduce the learning curve.

I need some advice about the book

Wed, 14 Jan 2009 17:15:23 GMT

I have got the final reviews about the book (Building Domain Specific Languages with Boo), and a few of them talk about a problem in the book that I am not sure how to solve.

The problem is, quite simply, the book talks about Boo a lot. I see the book not as a academic topic, but as something that would give the reader actionable knowledge. At the same time, I try to cover the entire life cycle of a DSL. So the book covers things like how to design a good syntax, documenting DSLs, how to structure them, versioning issues and other aspects that are mostly agnostic to the actual tool that we use to build the DSL.

On the other hand, things like integrating with a unit testing framework, or building UI, or the exact particulars about how to build certain aspects of the DSL and common recipes for frequent problems are strongly tied to the actual technology that we are using. Oh, we can probably discuss some of them in abstract, but I don’t think it would be nearly as valuable.

One of the goals that I have when writing this book is to create an actionable document, one that you can use to start working on a DSL right away. There are several DSLs out there that I can directly trace to the book (Simple State Machine and Horn), so I think that I did a good job there.

The problem, as reflected in the reviews, that people expected to see much more about DSLs and maybe get some brief discussion about the actual implementation in Boo.

I think that the main issue is different expectations that they had, and I am wondering what we can do about it. One of the options that we consider is to rename the book to reflect what it actually contain better. However, I don’t know what name would suit it better, and I don’t even know if that is the best solutions.

Can you offer some suggestions?

Should you be able to define new abstractions in a Domain Specific Language?

Tue, 23 Dec 2008 01:58:55 GMT

Fowler has a post about DSL, which contain the following:

The first is in language design. I was talking with a language designer who I have a lot of respect for, and he stressed that a key feature of languages was the ability to define new abstractions. With DSLs I don't think this is the case. In most DSLs the DSL chooses the abstraction you work with, if you want different abstractions you use a different DSL (or maybe extend the DSL you're using). Sometimes there's a role for new abstractions, but those cases are the minority and when they do happen the abstractions are limited. Indeed I think the lack of ability to define new abstractions is one of the things that distinguishes DSLs from general purpose languages.

I disagree with the statement that this is an exception. This is something that come up again and again in a DSL. You start with a given concept of what you want to do, and you give it to the users. Rinse & repeat a couple of time and you have a language that the users can start play with.

The main problem is that not giving the users that facilities for abstraction usually mean that they will work around your system, or that the DSL scripts that you will have will suffer from copy & paste, unnecessary complexity, etc.

A trivial example would be how to define customer roles. You want to be able to process orders based on customer role, and as such, it is important to be able to define how you choose a customer. Not only that, but even the customer selection criteria should be abstracted.

Here are a few examples:

Bad customer:

Bad credit rating
OR More than 5 returns last year
OR More than 5 helpdesk calls last 6 months

Silver Customer

More than 5 purchases

Gold Customer

Total purchases over 5,000$

Preferred customer:

Not a bad customer
Gold customer

We want to define roles using other roles. Because that is how the business thinks about it. We naturally create abstractions for ourselves, because this is how we think.

If we need to go back to the development team for each new abstraction, we have a heavy weight process that the users will work around.

Most of my DSL contains some form of user define abstractions, usually in the form of:

define bad_customer:
	customer.HasBadCredit or 
		customer.TotalReturnsIn(1.year) > 5 or
		customer.TotalHelpdeskCallsin(6.months) > 5

define silver_customer:
	customer.TotalPurchases > 5

define gold_customer:
	customer.TotalPurchasesAmount > 5000

define preferred_customer:
	gold_customer and not bad_customer

This is usually user editable, and we plug this into the intellisense of the system. In one project, I even provided a refactoring support, Extract Business Term (bound to CTRL+ALT+V, go R#!)

Abstractions are important, and building them into the language is just as important. You want to empower the users to extend your language, even if it a language that is very limited in scope. Because you give them the option to take the language to realms that you never dreamed on.

A very significant milestone

Mon, 08 Dec 2008 13:44:46 GMT

Today, around 3 AM, I finally submitted the last chapter in the book.

I was surprised to realize that it took over a year to write it, and I am happy beyond words that the action of writing the book is behind me.

That said, it is not actually the end of the work. All it means is that I am done writing new content, now we are going to get into a cycle of editing once more, and I really don't look forward to re-reading what I wrote five to ten times, over and over and over again.

Nevertheless, I am feeling GREAT!

Integrating User Interface & Domain Specific Languages

Mon, 08 Dec 2008 06:05:14 GMT

User interface is often seen as an external piece of the DSL, something that is completely separated from the language. While this is one options to handle things, it is not necessarily the only one.

During the design of the DSL, we should also think about the user interface that we intend to give our users. That will change the way that we build the language, usually, but not only because we need to give the UI more information, but because we can also utilize the UI to make things easier.

In particular, a good usage of the UI will allow us to give the users more information. Using the Shopping Cart DSL, we can utilize the UI to show the users what are the actions that will be taken on bad credit for all types of customers, or what it means to be a preferred customer. This is a very efficient way of writing documentation, once that is also guaranteed to be always updated.

A mockup of a screen reporting on the behavior of the system, as extracted from the DSL scripts:

This is a very important concept because it is showing the user data that the DSL isn't actually able to show. This is aggregated data across all behaviors and in all states, something that in the past you would have to either remember or research every time that you wanted to find out what is actually happening.

In short, don't try to limit yourself to just the typical IDE option, a DSL can and should give you a lot more information, and using this information can give you a lot of insight into your business.

The shopping cart rule engine DSL

Mon, 08 Dec 2008 05:35:29 GMT

As the final example in the book, I am showing off a DSL for processing a shopping cart. Here is how it looks like:

behavior of preferred_customer

upon bad_credit:
	authorize_funds cart.Total * 0.5,  "For preferred customers we only authorize half the amount"

upon cart_update:
	when cart.Total > 1000:
  		add_cart_discount 5, "Preferred members gets 5% discount for orders over 1,000$"

And this:

behavior of default_customer
upon bad_credit:
authorize_funds cart.Total, "We require full authorization of the amount in low credit rating scenarios"

Where we defined preferred customer and default customer as:

define preferred_customer:
	customer.TotalPurchaseAmount > 5000
	
define default_customer:
	customer.TotalPurchaseAmount <= 5000

The implementation is surprisingly easy, and I was able to walk through the reasoning for this implementation rather than the usual when [condition] in a way that I hope would make sense.

You can look at the implementation here: https://rhino-tools.svn.sourceforge.net/svnroot/rhino-tools/BDSLiB/trunk

Kazien Conf Workshops

Fri, 31 Oct 2008 15:35:28 GMT

Yesterday I gave two workshops, Advanced NHibernate and Building DSL with Boo. I finished the day absolutely crushed, but I think they went very well.

Both were recorded, although I am not sure when they will be online.

Ryan Kelley has a blow by blow description of the NHibernate talk, and you can get the code for that here:

https://rhino-tools.svn.sourceforge.net/svnroot/rhino-tools/trunk/SampleApplications/ORM+=2/

I'll post the code for the DSL talk shortly afterward.

I got some really positive feedback about the NHibernate Profiler, and I am very interested in demoing that and getting additional feedback when the real conference starts.

Every DSL ends up being Smalltalk

Sun, 26 Oct 2008 23:37:40 GMT

I had this though in my head for a while now. I built an IDE for a DSL, and somewhere toward the end of the first revision I understood something very interesting. My IDE wasn't actually using the textual representation of the language. The scripts that the user was editing were actually live instances, and they were fully capable of validating and saving themselves.

The IDE worked with those instances, used them to do its operations, and allowed to edit them on the fly. It was quite challenging to do, I must say, and I kept thinking about the image model of smalltalk, where everything is a live instance.

This brings to mind Greenspan's tenth rule, which state: Any sufficiently complicated C or Fortran program contains an ad hoc, informally-specified, bug-ridden, slow implementation of half of Common Lisp.

New version of the book is out

Thu, 02 Oct 2008 19:54:15 GMT

You can head to the Book Site and get a new version, because Chapters 1 - 11 (and appendix B) are available.

The only remaining chapters are #12 (DSL implementation patterns) and #13 Implementing a Real World DSL. Chapter 12 goes really well, so far, and I hope to have it ready by the end of the weekend.

Chapter 11 is done, or the tale of meta documentation

Fri, 26 Sep 2008 19:00:52 GMT

Here it the table of content:

Writing the Getting Started Guide

Create Low Hanging Fruits

The User Guide

Documenting the Language Syntax
The Language Reference
Debugging for business users

Creating developer documentation

Outlining DSL structure
The syntax implementation

Keywords
Behaviors
Conventions
Notations
External Integration

Documenting AST transformations
Executable documentation
Summary

The chapter starts with...

Documentation is a task that most developers strongly dislike. It is treated as a tedious, annoying task and often falls on the developer who protests the least. One additional problem is that a developer trying to document his own work is often not going to do a good work.
This is not an aspiration against developers in general; the problem is that there are too many things that people who actually write the code takes for granted. And even in we ignore that, developers tend to write to developers, in a way that makes little sense to non developers.

That is true for me as well. And trying to document how developers should write documentation is... hard.

At least I can look forward for a really interesting Chapter 12.

Enter the demoware

Sun, 21 Sep 2008 08:49:55 GMT

I am writing about documentation at the moment, and I found myself writing the following:

I don’t think that I can emphasize enough how important it is to have a good first impression in the DSL. It can literally make or break your project. We should make above reasonable efforts to ensure that the first impression of the user from our system would be positive.
This includes investing time in building good looking UI, and snappy graphics. They might not actually have a lot of value for the project from a technical perspective, not even from the point of day to day usage in some cases, but they are crucially important from social engineering perspective.
A project that looks good is pleasant to use, easier to demo and in general easier to get funding for.
This also includes the documentation, if we can do something in a short amount of time; we get a level of trust from the users. “Hey, I can make it go bang!” is important to gain acceptance. The first stage should be a very easy one, even if you have to design to enable that specifically.

After reading that, I quickly added this as well:

Note, however, that you should be wary of creating a demoware project, one that is strictly focused on demoing well, and not actually add value in real world conditions. Such projects may demo well, and get funding and support, but they tend to fall into the land of tortureware very rapidly, making things harder to do, instead of easier.
Beware of the demoware.

Persistent DSL caching issues

Wed, 10 Sep 2008 15:24:16 GMT

A while ago I talked about persistent DSL caching. I was asked why my solution was not a builtin part of Rhino DSL.

The reason for that is that this is actually a not so simple problem. Let me point out a few of the issues that are non obvious.

Need to handle removal of scripts
Need to handle updating scripts
Need to handle new scripts

Those are easy, sort of, but what about this one?

Need to handle DSL updates

When you are in development mode, you really need to know that changing the way the DSL behaves would also invalidate any cache.

I like to keep a very high bar of quality on the software I make, and there is a fine distinction between one off attempts and reusable ones. One off attempts can be hackish and stupid. Reusable implementations should be written properly.

And no, there isn't anything overly complex here. Just time to test all bases.

Anyone feels like sumbiting a patch?

Implementing generic natural language DSL

Mon, 08 Sep 2008 19:27:16 GMT

I said that I would post about it, so here is the high level design for generic implementation of natural language looking parsing. Let us explore the problem scenario first. We want to be able to build this language, without having to build a full blown language from scratch:

open http://www.ayende.com/
click on link to Blog
click on link to first post
enter comment with name Ayende Rahien and email foo@example.org and url http://www.ayende.com/Blog/
enter comment text This is an awesome post.
click on submit
comment with This is an awesome post should appear on page

And to prove that we are not focusing on a single language, let us try this one as well:

when account balance is 500$ and withdrawal is made of 400$ we should get a low funds alert
when account balance is 500$ and withdrawal is made of 501$ we should deny the transaction
when weekly international charge is at 3,500$ and max weekly international charge is of 5,000$ and new charge arrives for amount 2,230$ we should deny the transaction

I think that those are divergent enough to show that the solution is a generic one.

And now, to the solution. Each type of language is going to have its own DSL engine, which know how to deal with the particular dialect that we are using. The default parsing is a three steps solution. First, split the text into sentences, then, split each sentence to tokens by whitespace. Now, for each statement, we search for the appropriate statement resolve, which is a class that knows how to deal with it. The statement resolver methods are then called to process the statement.

There are two key principal to the design. First, turning something like 'click on link' to an invocation of the ClickOnLink statement resolver and lazy parameter evaluation.

This is going to be interesting, the time right now is 19:38, and I am going to start implementing this.

It is now 22:04, and I finished the first language.

Working on the second now. It is 22:10 and I am done with the second one.

What did I do?

I took the text we had and turn that into executable commands. Now, this isn't flexible at all. If you make a modification in the way it is structured, it will fail, coming back to why natural language is a bad choice here, but it had quite a bit of flexibility in it.

You can get the code for this, including tests, here: https://rhino-tools.svn.sourceforge.net/svnroot/rhino-tools/experiments/natrual-language

But let us talk for a bit about how this is implemented. I'll show the bank example, because it is easier.

We start by defining the BankParser, which looks like this:

The bank parser merely define what the statement resolvers are, and any special parsers that are needed (in this case, we need to handle dollar values).

A statement parser is trivial:

And yes, those are pure POCO classes.

The whole idea here was that I can implement some smarts into the default engine about how it recognize methods and resolve parameters. I will admit that overloading caused some issues, but I think that this is pretty simple implementation.

It also does a good job in demonstrating the problems in such a language. Go ahead and try to build operator precedence into it. Or implement an if statement. You really can't, not without introducing a lot more structure into it. And that would turn it into yet another programming language.

What about the tooling? Intellisense and syntax highlighting?

Well, since we have the structure of the code, and we know the conventions, you shouldn't have a problem taking my previous posts about this and translating them directly into supporting this.

And yes, I can create a language in this in a few minutes, As BankParser has proven.