http://ayende.comAyende @ RahienCopyright (C) Ayende Rahien 2004 - 2021 (c) 202660That's the problem in a nutshell and why the breakingpoint / threshold is so high. With ILOG you basically send two programmers off to training and when they get back by and large no one can talk with them anymore and they can't explain what it is they do. This is where IBM, Oracle, and MS need to step up to the plate and put some effort into making these systems more accessible, usable, and easier to manage and deploy. Absent that the threshold will remain high and lesser solutions will still need to be developed and employed. It is such a ubiquitous requirement however in systems of any note at all it seems crazy to me that we'd enter a "SaaS" / "PaaS" world without vendors doing some serious investing in critical aspect of any such platform. In the end, a part of enterprise systems of any complexity will always be a nightmare until this imperative is addressed. http://ayende.com/3548/jfhci-considering-scale#comment24http://ayende.com/3548/jfhci-considering-scale#comment24Sun, 24 Aug 2008 02:21:16 GMTWhy is it so important to cut the developer out of the loop? http://ayende.com/3548/jfhci-considering-scale#comment23http://ayende.com/3548/jfhci-considering-scale#comment23Sat, 23 Aug 2008 17:47:40 GMTCasey, Traditional rules engines suffers from a very simple problem. a) they tend to suck b) they require a dev anyway for anything beyond the truly trivial. The concept of BA can change the rules is fairly easy to achieve, you can build the DSL for that in an hour. The problem then becomes the ability of the system to manage that, and _that_ is hugely expensive. http://ayende.com/3548/jfhci-considering-scale#comment22http://ayende.com/3548/jfhci-considering-scale#comment22Sat, 23 Aug 2008 15:20:20 GMTWhen you beyond the simple, then move to a proper rules/decision engine - don't hard code. Hard coding says "only a C# developer can ever change your rules" and this is exactly what we should be trying to avoid. I want a system that says "any business analyst can change your rules" Hard coding is a "cheap" solution - and if that is a business requirement then all well and good - but know the business implications and costs first. More than a little concered at some of the comments on here which sound very much like job protections schemes. If you have done a good job on a code base you are then redundant as far as that code base is concerned. A bad job requires you to be involved afterwards. http://ayende.com/3548/jfhci-considering-scale#comment21http://ayende.com/3548/jfhci-considering-scale#comment21Sat, 23 Aug 2008 11:26:29 GMTLucas, RSS is distributed polling. What I am talking about is centralized polling. See Twitter for the problems this can cause http://ayende.com/3548/jfhci-considering-scale#comment20http://ayende.com/3548/jfhci-considering-scale#comment20Fri, 22 Aug 2008 22:42:45 GMT@Oren Kind of off topic, but how so with the Pull systems not scaling? It seems RSS feeds scale very well and I thought the whole point of nService bus (and others of that ilk) was to allow for better scaling. Perhaps I'm using the wrong terminology for what these systems are... (wouldn't be the first time) http://ayende.com/3548/jfhci-considering-scale#comment19http://ayende.com/3548/jfhci-considering-scale#comment19Fri, 22 Aug 2008 22:26:47 GMTI've seen insurance, taxation, and permitting apps with thousands of rules with exceptions to nearly every rule - in these systems configuration, hard coding, and admin forms just don't cut it. There just gets to be a breaking point where only a [complete] rules environment / system will suffice. The issue seems to be one more of what to do below that breaking point. My own opinion all along has been that rules is a tough problem space and been sort of a 'final frontier' in the software development world such that the breaking point has always remained high with those above it served by ILOG and everyone below it rolling their own [painful] response in some fashion or another. http://ayende.com/3548/jfhci-considering-scale#comment18http://ayende.com/3548/jfhci-considering-scale#comment18Fri, 22 Aug 2008 20:25:58 GMTPull system tend to suffer very bad scaling issues http://ayende.com/3548/jfhci-considering-scale#comment17http://ayende.com/3548/jfhci-considering-scale#comment17Fri, 22 Aug 2008 20:12:54 GMT@Ayende, One thing not being discussed is that MANY of these rules (notifications in particular) can be removed completely from the system and replaced with pull systems (like Rss Feeds/nService messaged/etc) instead of push systems (email notifications). I've found some of the most complex business rules in our systems are for notification emails. I don't mean "action" emails like "approve this", but general "someone has done something that you may be interested in knowing about". Users appreciate the focus shift as well. So going hand-in-hand with the idea that "Does my customer really ever need to do this" on customizing rules and base values is the idea "Does my customer really need ME to do this". If people are truely honest about this, they find VERY seldom does all that customization actually need to be built, and usually when it does you're better off writing an extensible system for developers, not for the general users. http://ayende.com/3548/jfhci-considering-scale#comment16http://ayende.com/3548/jfhci-considering-scale#comment16Fri, 22 Aug 2008 20:03:23 GMTJoe, No, I DON'T want to make this configurable. That is the whole point. There is a huge cost of making things configurable http://ayende.com/3548/jfhci-considering-scale#comment15http://ayende.com/3548/jfhci-considering-scale#comment15Fri, 22 Aug 2008 19:49:54 GMTI like that I can see how this can become a business decision now. A higher cost employee that can turn up a new feature in a fraction of the time another one would I find myself compelled to go research smalltalk now as this scenario seems familiar. http://ayende.com/3548/jfhci-considering-scale#comment14http://ayende.com/3548/jfhci-considering-scale#comment14Fri, 22 Aug 2008 19:44:13 GMTTrue. I often find a need to establish a middle-ground, though, where the simplest and most common values are easily editable by an administrator (or even a standard user, depending on where the rule is applied). For the majority of values, though, JFHCI. http://ayende.com/3548/jfhci-considering-scale#comment13http://ayende.com/3548/jfhci-considering-scale#comment13Fri, 22 Aug 2008 19:10:34 GMTTyler, Sort of. What I am saying is actually that you don't want to provide admin interface for the majority of the rules values, because it is a problematic issue http://ayende.com/3548/jfhci-considering-scale#comment12http://ayende.com/3548/jfhci-considering-scale#comment12Fri, 22 Aug 2008 19:05:02 GMTAyende's right on the money, as usual. Hard coding rules and rule logic improves maintainability and reduces time-to-market for those rules. This does not mean that one cannot provide an admin interface for the __values used in rules. Yes, in order to create new rules developers must be on hand, but this would be necessary for anything but the most trivial of rules anyhow. http://ayende.com/3548/jfhci-considering-scale#comment11http://ayende.com/3548/jfhci-considering-scale#comment11Fri, 22 Aug 2008 19:00:36 GMTSeems like you're doing a lot of fishing around, and looping back hard onto, the data and code extensibility issues. I will say your choice of examples begs a question I feel all the main vendors have dodged for years - standalone rules engines, i.e. 'MS Rules Server 2009'. We now have the ability to use WWF rules without workflows, but that's not the same as having a full featured VS-like environment for rules development, deployment, and management. Hopefully this will be something that emerges from OSLO and won't be buried deeply in the mix way it has been to-date. My understanding from folks inside of IBM, MS, and Oracle is that the problem they've had addressing this over the past decade is largely the same at each and primarily revolved around turf issues. MS has OSLO, IBM just picked up ILOG so their direction is clear, not sure what Oracle has had going on of late. http://ayende.com/3548/jfhci-considering-scale#comment10http://ayende.com/3548/jfhci-considering-scale#comment10Fri, 22 Aug 2008 18:42:38 GMTCasey, You seem to be missing the point I am making. Probably my fault for not choosing a better example. I am not talking about hard coding stuff like base values for tax calculations. Certainly, for things like that, you need a way to deal with this. I am talking about hard coding the rules that work on top of that, and not exposing all the rules parameters. Again, I am talking about using this approach to facilitate change, easily. No, it is not a replacement for configuration and not a replacement for the database. But it is a great approach to prepare for change. http://ayende.com/3548/jfhci-considering-scale#comment9http://ayende.com/3548/jfhci-considering-scale#comment9Fri, 22 Aug 2008 16:10:29 GMTJust tell us, when we'll have Rhino.Rules? http://ayende.com/3548/jfhci-considering-scale#comment8http://ayende.com/3548/jfhci-considering-scale#comment8Fri, 22 Aug 2008 15:26:27 GMTWhat are your thoughts on using a Workflow/Business Rules engine to do what you're talking about? It's somewhat the best of both worlds. You don't have to create a bunch of UI elements to support rules changes. And you don't have to go all the way down to code to make the change. In theory a business user could even learn how to use it. I guess that's the same scenario your DSL script would enable. http://ayende.com/3548/jfhci-considering-scale#comment7http://ayende.com/3548/jfhci-considering-scale#comment7Fri, 22 Aug 2008 14:43:04 GMTI like the hard code approach as there is nothing to stop me from hard coding a rule that talks to storage that is updated by a UI, allowing the rule to be changed when required. Rather than making all the rules editable. http://ayende.com/3548/jfhci-considering-scale#comment6http://ayende.com/3548/jfhci-considering-scale#comment6Fri, 22 Aug 2008 14:37:33 GMTSome of the examples you are gving are clearly "you are gonna have to hire a developer for this", but many are "is there a business requirement to be able to change this rule?" The second for example is ommon in accounting systems, where rules and formulas for things like sales tax are clearly defined and rarely change, but the figures that are used, and the items they apply to changes at least every year if not more often. A long while back I worked for one of the world's largest accountancy software suppliers, and they literally had to have developers standing by on budget day to rewrite large components of their software the night budget changes were announced, and to ship out disks to their customers the next morning. This is the result of the scenario you are describing. A later "evolution" of that software was to make a large number of those calculations depend upon externally defined (via a user administration section) values. They no longer had to have so many devs standing by. I appreciate YAGNI - but to me that is a chance to call your business owners out ... ask them to justify the requirement for change as a request (user story/whatever) and then give them the cost for letting them change it. They can then make th decision. As a buiness owner, I wouldn't want to *have* to have C# developers around just to change things when I wanted a rule changed, and to be honest most DSLs are worse. Either give the users a simple admin screen to alter these kind of values, or a simple admin screen that can compose formulas or decision rules ... or identify that a developer changing it each time is more cost effective (and for that it would have to be very infrequently changed and horribly complicated to express in an admin screen) Certainly shy away from XML, but job preservation is something I'm not in favour of ... the sooner computers do get smarter and developers can stop writing code and can start expressing business requirement the better. http://ayende.com/3548/jfhci-considering-scale#comment5http://ayende.com/3548/jfhci-considering-scale#comment5Fri, 22 Aug 2008 14:36:24 GMTAyende, Agreed. It's when you need to have a balance between the two that you would start using one. But you bring up a good point - maybe there's a need to have a rules-engine which is capable of some of the things you mention. Of course, as you aptly pointed out, for truly complex decision trees, you are going to have to have everyone involved anyway. But still might be fun to poke around with. http://ayende.com/3548/jfhci-considering-scale#comment4http://ayende.com/3548/jfhci-considering-scale#comment4Fri, 22 Aug 2008 14:25:19 GMTPoint is that whether you provide an admin tool or a DSL, it is irelevant. You need to make this configurable. Also the support line applies to both cases http://ayende.com/3548/jfhci-considering-scale#comment3http://ayende.com/3548/jfhci-considering-scale#comment3Fri, 22 Aug 2008 14:19:32 GMTCory, Rule engines commonly suffer from a few major problems. One, they tend to be much harder to work with than code for developers. Two, they requires developers to work in any non trivial scenario. Three, source control. http://ayende.com/3548/jfhci-considering-scale#comment2http://ayende.com/3548/jfhci-considering-scale#comment2Fri, 22 Aug 2008 14:18:41 GMTAh, I get it more now. In the Java world, we've used things like JRules (http://www.ilog.com/products/jrules/) before. We've also done the assembly based (well, jar based) rules-in-code way, for a very complex system. For those scenarios, you can't do an admin interface. For example, our system looked at a set of records and based on business rules and filters dynamically removed or create new virtual records to be passed on to the rest of the system. These rules were insanely complex (If record is from Florida and the title is reported as Michigan and the date was between 6/2003 and 9/2003 and we've seen an earlier record that has a possible event, then change this record type to be from Florida but marked as Baz), and were *not* easily adjustable from the business. I think the problem was that the example of discounts is deceiptively simple, and putting it in the context of scale clarifies that. What we really need is a Rules Engine for .NET (looks like there are a couple out there, I'll have to look more into them) http://ayende.com/3548/jfhci-considering-scale#comment1http://ayende.com/3548/jfhci-considering-scale#comment1Fri, 22 Aug 2008 14:12:04 GMT