AnswerDon't stop with the first DSL abstraction
The problem as it was stated was of rules that looked like this:
upon bounced_check or refused_credit: if customer.TotalPurchases > 10000: # preferred ask_authorization_for_more_credit else: call_the cops upon new_order: if customer.TotalPurchases > 10000: # preferred apply_discount 5.precent upon order_shipped: send_marketing_stuff unless customer.RequestedNoSpam
I don't like it, and the reason isn't just that we can introduce IsPreferred.
I don't like it because the abstraction facilities here are poor. We have basically introduced events and business rules, maybe with a sprinkling of a domain model, but nothing really meaningful. Such system will die under their own weight in any situation of significant complexity (in other words, in all real world situations).
Let us consider the problem in reverse, shall we? We have various conditions and actions upon which we can act. But the logic is scattered all over the place, making it hard to read, modify, understand and work with. When such a system compose of the lifeblood of the business, the business usually adapts, and starts to talk in the terms of the system. However, they tend to lose the ability to think about things in way that would be more meaningful.
I listened today to a business person trying to explain some concept that he wanted to make. It took him several tries to explain the business problem because he was focused on the technical one. The system has a corrupting affect on it. I call this the Babel Syndrome, the reverse of DDD's ubiquitous language.
Let us see if we can get a high level of meaning out of the above DSL, shall we? First, we restate our problem, instead of dealing with events and conditions for responding the events, we deal with business responses for scenarios. It doesn't sound like much of a difference, but in actuality, there is a big difference between the two.
The most important of those differences is the change from handling the events to handling a business scenario in a given context. In other words, instead of asking what we should do when a check is bounced, we need to ask a totally different question. "When the customer is preferred, what should the response be for bounced check?"
This is anything but a minor change in the the way we think about the language and how we operate on it. Let us see the DSL script, after which we can discuss how it affects us. These are the contents of the default.boo file:
upon order_shipped: send_marketing_stuff unless customer.RequestedNoSpamupon bounced_check or refused_credit:
This will be executed for all orders, like before. Now, let us look at preferred_customer.boo, and what concepts it express.
when customer.TotalPurchases > 10000 # preferred upon new_order: apply_discount 5.precents upon bounced_check or refused_credit: ask_authorization_for_more_credit
And now we are getting to see some of the more interesting parts of the difference. We are now talking in terms of a business scenario. When we have a preferred customer, and something happen, how should we respond?
This change is a well known refactoring: conditional to polymorphism. In other words, we just created the strategy pattern with a DSL. The difference here is that the script have an active role in deciding whatever it can deal with the scenario or not (in other words, chain of responsibility, and the pattern I am going to mention).
When we need to handle some business scenario, we are going to execute all the scripts, with the default.boo being the last one to run. If any of the scripts accepted the scenario as valid and has specific action to take, it has the option to do so.
Enough about the implementation, let us go back to the concepts. We can make now talk to the business people in a way that is far more concise and natural. Instead of having to focus on all permutations of a possible event, we can now talking about a specific scenario and how we handle the business event in that context. Not only is this more readable, it is easier by far to actually define such things as what is the meaning of a preferred customer. I can open the DSL and actually read it.
Similar approaches are very useful when you recognize that the code is asking to be given a more explicit shape than just generic rules. Don't let your DSL be whatever you started with. Find and actively extract higher level meanings whenever it is possible.
A deeper examination of this DSL, how to build and use it is likely to compose most of chapter 13, as a real world example of a complex DSL. Who do you think?
Given this approach, how would you design an offer management DSL?
More posts in "Answer" series:
- (05 Jan 2023) what does this code print?
- (15 Dec 2022) What does this code print?
- (07 Apr 2022) Why is this code broken?
- (20 Jan 2017) What does this code do?
- (16 Aug 2011) Modifying execution approaches
- (30 Apr 2011) Stopping the leaks
- (24 Dec 2010) This code should never hit production
- (21 Dec 2010) Your own ThreadLocal
- (11 Feb 2010) Debugging a resource leak
- (03 Sep 2009) The lazy loaded inheritance many to one association OR/M conundrum
- (04 Sep 2008) Don't stop with the first DSL abstraction
- (12 Jun 2008) How many tests?
Well this approach also has a very good visualization.
You can display a table with contexts as rows and events as columns and specify corresponding actions in the cells. So row "Is Preferred", column "New Order" will contain "Apply Discount 5%".
This graphical representation will also give you an instant hint if you miss some important rules.
So now we have inheritance in DSL. That's nice, I like the approach but there should be a rule preview and perhaps something like the import or reject statement so we can easily choose inherit from a default set of templates or not. Also a rules review panel so we know exactly whether we missed something or accidentally override the other.
so would you then also have another file for those customers who have asked for too many credit extensions as well?
when customer.CreditExtensions > 20 # risk
upon bounced_check or refused_credit :
the problem that i see is that now you have to either rank the scenario or setup some sort of condition catching...
classify #preferred when customer.TotalPurchases > 10000
classify #risk when customer.CreditExtensions > 20
upon bounced_check or refused_credit :
and i personally don't like that route either. as you can tell you will end up with a conditional nightmare. perhaps something like:
when customer.TotalPurchases > 10000
when customer.CreditExtensions > 20
upon bounced_check or refused_credit :
i hope i making sense...
What you're talking about in terms of "scenario" or "business scenario", I talk about as "context" in Context/Specification styled testing.
The goal and the result is similar: stop talking about implementation of the software, and start talking about the experience - either of the business or the software.
Yes, I would have a separate file for that.
I would try to build it in such a way that ordering doesn't matter. Because quite often, while this an edge case that would bother me as a developer it is just not a real scenario
Can you be both preferred customer who is often asking for credit extensions, maybe... but then I would have a special treatment for this scenario.
Another option would be to setup priority for everything. So preferred customer is at priority 10, and if it run, then nothing else does, but if it dosn't, lower priority stuff get to run.
I really don't like the idea of conditions, I don't see how it is any different than the problematic scenario that we had before, which we want to escape.
Having polymorphism instead of conditionals seems like a nice goal. But polymorphism is only simple when using single inheritance. In your example, you're replacing a linear inheritance hierarchy with arbitrary conditions. What if several rules apply to the same transaction, and what if they "override" the same rules?
From a software developer's perspective, I'd go out of my way to keep the modularity and think hard about ways to explicitly deal with conflicts. (In fact, we implemented that with our mixin library, where several mixins can overload the same method, and this gets resolved by just chaining base-calls up some non-arbitrary order. Works fine, but it does require some abstract thinking.)
I'm not sure if your DSL is expressive enough to let me do this. Also, given enough complexity, you're going to need some kind of x-ref tool that tells you what gets actually executed, and in which order.
If you want business analysts to deal with this DSL, I'm quite sure they are more comfortable with conditions in rules. At least that makes the execution of any rule clear to the reader. It makes conflicts obvious, and they can be avoided easily. Not beautiful, true, but how far do you want to take both a single-purpuse language and its users?
Oh, I just noted that you already went through this with meisinger.
You seem to think that this is a rare case. My guts say that this won't be generally true, but it's hard to argue with that if there is no data to look at. I'd say go for it, try it in a real project. When a set of rules is large enough to break down with linear conditions, and still conflicting rules do not present that much of a problem, you win.
While it'd be hard to generalize such an experience, it would still be quite interesting.
x-ref tool is easy to do, at any rate.
And having several scripts responding to the same event is actually simpler to implement than selecting the appropriate one.
If you really want that, you can do something like:
when customer.TotalPurchases > 10000
Which would be very easy to resolve.
Sure, but when a rule is pseudo-polymorphically distributed among several files, and you essentially have to search all files for this rule or use a tool, it's hard to see the conflicts, or to know how to assign priorities or ignore-directives. (ignore does not solve conflicts of more than two rules that can occur in any combination, btw.) On the other hand, if all conditions for one rule are in a single block, it's easy to tell what happens, and to modify it. I agree that this is not a good software engineering practice, but this is not exactly the same game.
Executing all rules might work in some cases, but even then it presents the problem of ordering (unless your rules are side-effect free)
The only way to tell if this works is to try it, though.
a) in general, you don't want to see the conflicts. This is a mater of system behavior. The rules are not side effect free, by design (calling the cops is a side effect :-) ).
However, they can be set not to immediately execute, so you get an idea about system behavior before it is actually executed. And can make certain checks there.
b) providing good visualization about what is going to happen at each action for all the rules isn't hard, and can be done as part of editing the scripts in the UI.
c) All conditions for one rule are in a single place is a problem. Not because it is not good software engineering, but because you are now asking the business user to think about things in a way that is most unnatural.
Let me try to explain, go ask a business user something like: "What is a preferred customer?" vs. "What should we do for large orders?"
In the first case, they usually have a very clear and concise answer, backed by the business strategic vision, contracts and marketing information.
For the second, they have to dredge up all the possible conditions.
A bit off topic but then again, not if I think of it: I don't know if you address this issue in your book, but when I look at those rules, there's one thing what I find difficult - in large rule based systems you will eventually need the meta level i.e. a possibility to have some sort of abstract form to describe the rules so you can analyse and dissect the rules not just process them. The reasons for this are different, from conflict detection to usage analysis and visualisation. While DSL can be most expressive, the expressiveness of the language is in inverse proportion to its "dissectability". Introducing new concepts using objects makes this analysis a lot easier (in addition that people tend to wrap their heads around objects more easily) . So eventually, you will reinvent just a good OO design in your DSL... Hmm, was there a point in my comment, figure out I must.
Amazing, I did something about that two days ago, and wasn't sure if I wanted to post that or nto.
how you acquire the knowledge you need to build those rules is one matter. But what do you do with this information?
You go for the executable specification. The way it flows out of interviewed users mouths is how it's written down, and then you feed it to a machine that can execute it. This does have many advantages, granted.
The option that you already dismissed is to take the information you get from your users (grouped by condition) and manually transform it into another structure (by event, with all conditions manually assembled for each event).
The question is only, how much thought has to go into this assembling? Can it just be merged automatically? Does the user have to give hints (priority, ignore)?
What is more confusing? Keeping the the specs and the implementation separate (with all those well-understood problems), or keeping it together and having to do the merging in your head (or visualized by a tool) each time you need to understand how a specific case would be processed?
Which is easier to test? How can I know which FIT tests to write without seeing how many different code paths a single event might follow? Which combinations of input data will give me full coverage?
I'm not saying you're wrong. It might work. But I think your polymorphism analogy breaks here. In C#, a refactoring that replaces if loops (ha!) with polymorphism is a good thing and almost always brings more advantages than disadvantages. What I'm saying is that from this you cannot conclude that the same is true for what you're proposing. It's not automatically better by inductive reasoning.
It's nothing that we can know in advance, which makes it quite interesting. Experience will tell.
What you've written in a) I completely did not get. Sure there are side effects in real live. That's why I think that just executing all possible rules, in any order, will not work.
The option I dismissed was to keep thinking in conditional on the events.
I don't want that.
Being able to see exactly what will happen in the case of an event, including everything that goes on across all variations is essential. That is understood and agreed.
BUT, that isn't how I want to build and think about the system.
Also note that I am not actually saying anything about how I am going to implement it, only that the way that I think about it should be scenario based, not conditional on event based.
The problem of conditional on event is that it fractures the logic, make maintenance harder and force you to think in unnatural ways.
If the only thing that is an issue with the scenario based approach is that you want to see everything that goes on for an event, that is very easily done.
How to test? You test each scenario in isolation.
Then you go to the business and ask them about more complex scenarios, what happen when a check bounce for a preferred member for the 10th time?
Something else that I think you should consider is that the output of this isn't the actual execution. It is the instruction to execute. And we can build a set of rules that manages that.
For example calling the cops and extending credits are invalid together.
(Totally Off Topic - I am using Google Chrome to write this comment and it is totally sweet...)
i guess i am getting hung up over the fact that i (as a simple developer) don't know how to ask the right questions
(if i did then why would i need a DSL, right?)
perhaps that is what i was missing... or maybe i still am :)
i was looking at this as something that the business user will use or create (in the spirit of crossing the great divide)
in other words... i was thinking that a business user would sit down and create a file called "preferred_customer.boo"
am i correct in this thinking or are you coming from the perspective that the business user will be using a UI to create these scenarios?
if we are talking about a file named "preferred_customer.boo" that is generated or created from a UI then i get it... 100%
a UI could totally handle the presentation and management of scenarios in such a way where the business user gets the "50,000 foot view"
an entire view of the scenario and events is not important to the system only to the business user
if we are talking about a business user generating this file then i think that my earlier post and Stefan's points hold more value since again the entire view is more important to the business user
Checked the stats, almost 8% of the visitors are using Chrome.
I don't have a UI in mind for that. That is not required.
But at any rate, if we had UI, it would be, at most, something like a syntax highlighted text box and a view of the relevant sections of related scenarios
Hey, I totally agree that this would be sweet if it works, and I also don't want to spend my life in linear condition evaluation mode if it can be helped. I'm only not quite as sure that the automatic merging of several rules is so easily manageable. If the user has to give hints, visualization tools are helpful but it still has to be done. This requires quite some abstraction and discipline.
If you ever build something like that for an actual system, tell us how it works out!