Ayende @ Rahien

Refunds available at head office

A vision of enterprise platform

A while ago I asked what kind of constraint an enterprise platform should deal with, I have been thinking about this ever since. Let us go over the list of constraints that have been brought up, and then I am going to discuss how I think about solving them.

  • Extensible in an easy manner - note that this holds for business analysts and for developers, both are groups that are likely to do work on the system. Ideally we can have some sort of a common interface that would make both people happy.
    • New entities
    • User Interface:
      • Forms
      • UI elements
      • Editing existing forms
    • Replacing core services
  • Upgradable - We want to allow the users to move from version 2.0 to 3.0 without having to re-write everything. This means that we need to make clear what we allow the user to do to the system if they want to have a successful upgrade.
  • Auditable - I don't suppose that I have to explain why, right?
  • Performant - well, it should. Considering the other contestant, that could be a major selling point.
  • Scalable - I want to be able to scale wide, so the CRM can scale as I add more servers.
  • IT friendly - expose state in a way that makes sense to the admin, allows monitoring and tracing easily, dashboard, interactive console.
  • Scriptable
  • Security - both for features and for the data
  • Data integration - allow to easily get and extract data from the system, including in bulk.
  • Hostable - can be run in a datacenter, can have multiple instances on a single machine securely
  • Easily Migratable - Data/Plugins/Code needs to be easily migratable/promotable from Development to Test to Production
  • Easily Comparable - I know my Production and Test environment are exactly the same (except for the data)... aren't they?
  • Exportable/Importable - I'll back it up to a file, import it onto a new server. Change a few settings like hostname and connection strings and have a clone site for whatever nefarious purposes.
  • Xcopy Deployable  - Well, as much as possible anyway
  • Easy Backup/Restore process
  • Customizable layout

I should point out, again, this is a theoretic exercise, but I am willing to stand by the assertion that I am making here. Everything is both possible and not too hard.

In addition to the above mentioned constraints, there is also my post about Evaluating a Business Platform, the list of requirements from this post has some duplication with the constraints, naturally, but they also deserve a mention:

  • Source Control - should be easy, simple and painless.
  • Ease of deployment
  • Debuggable - easily
  • Testable - easily
  • Automation of deployment
  • Separation of Concerns
  • Don't Repeat Yourself
  • Doesn't shoot me in the foot
  • Make sense - that is hard to explain, but it should be obvious what is going on there
  • Possible to extend - hacks are not something that I enjoy doing

I am going to just go through the solution I have in mind, and point out how it answers the constraints above.

First of all, we are talking about a web application, that is much easier than a client/server system, although that is an option as well. As such, I am going to use my default architecture here. This means that in terms of technology, we are going to be based on NHibernate and ActiveRecord for data access, MonoRail for a web framework, Windsor as a core concept and Binsor to configure it.

Why this stack? Because it is familiar to me, performant, extensible, easy to scale from the simple demo app to a complex enterprise applications and promote good practices.

The main concepts are change management (source control), distributed deployment and a tiny core that can be extend upon by the users. The core handles such things as managing the application state and displaying the UI, etc. Anything else, including all the core services are handled via extensions. From a deployment perspective, here is what I envision:

image One of the more important ideas is the ability to run multiply instances of the application on the same machine.

This ensure that it is a developer friendly application, not one that attempts to take over the entire machine.

The diagram paints the extreme case, obviously we can collapse all of those into the same machine.

You note that caching is built directly into the deployment design, this is because I intend to make heavy use of it for making the application scalable.

Another interesting thing to note is the SCM server as an underlying concept. Deployment in this application is done by committing a change to the /production branch.

This gives you some nice benefits out of the box, auditing all the changes in the application, for one, as well as very easy way to push changes to production, using well known and debugged tools. It also means that it is trivial to work in a team and that branching is not an issue.

The ASync server is to handle long running tasks, from batch processes to workflows to ETL processes.

Each application gets two databases (probably more, as a matter of fact, I skipped a logging & auditing DB, which I like to keep separate). One is for the application to manage itself, the second is for the actual data that you put inside.

I am not sure about the read only copies, though. Probably this is something that is better off handled by the database itself, using master/slaves arrangement.

I have started from the deployment diagram on purpose, because it is a good overview of the concepts that we are having. Source control and auditing are checked. Scalability through throwing cheap machines at it, checked. There may be some issues about that database, but this is why I introduced caching at this level already. Hostable is also checked, we can run several of those on a single machine. Easy migration is also checked.

XCopy deployment... well, that is a bitch in such a setup, but for a new machine, it should involve getting the code there and running deploy.cmd, not messy setup processes. Backup & restore, we basically need to backup the SCM and the DB, nothing more, so that shouldn't be a problem.

Let us talk a bit about what I mean by putting SCM in the deployment diagram, okay? Here is the structure that I have in mind right now, but remember that this is just speculation at the moment.

image

What is the point in this?

This layout pretty much mirrors the main extension points that I have for the application. I am still undecided about this structure, for several reasons, but let us go through how it works in order to explain what are the problems that I have with it.

The views, controllers and entities are pretty much self explanatory in their purpose, if not their implementation. As I mentioned, except the core, everything is an extension. Let us say that I want to go and define the Account entity. A screen will be provide by the core application for creating and editing entities. This screen will provide UI for defining a new entity, defining new fields and defining their UI characteristics.

This screen will be responsible for generating three files. /entities/Account.boo, /controllers/AccountController.boo, /views/Account/edit.brail

These files represent the way the sum total of information that the system needs to know about an entity. Those are also versioned files, and more importantly, they are very malleable for user modifications.

If the developer prefers to use C#, they can build a project which will contain Account and AccountController classes, and place the resulting assembly in the /binaries folder. In both cases, the application will recognize and load the new code on the fly. This is critically important to ensuring developer productivity and happiness, I have found. If external projects are used, they should be stored in the /src folder.

Obviously, both the boo files and the compiled binaries can also use other assemblies that are located in the binaries folder.

The usage of explicit controllers means that you are free to put whatever logic you want into the process, as well as modify the view to contain additional UI elements that can use to express logic better. Persistence is easy, because using NHibernate and ActiveRecord makes this trivial, but this also points out to an important issue, we can now access the data from the CRM in a very easy fashion, using all the power that NHibernate gives us, which is far from trivial.

Now, what goes into the /tasks and /workflows folders?

Tasks are scheduled tasks, occurring at specified intervals to execute some sort of a business operation (canceling all withstanding orders that are on hold for more than 30 days, for instance). Workflows are just that, a way to specify actions to happen in a more global fashion, using a DSL. A trivial example may be:

on case:
   when case.IsSolved == false and case.DueDate > CurrentTime:
        raiseAlert "Should have handled ${case.Id} by ${case.DueDate}, but the case is still ${case.Status}."

This is also a way for a business user to extend the system. A UI layer for that is probably also in order.

The /config contains the configuration of the application, basically the binsor configuration and maybe the web.config files. Using that, you can override the default services of the CRM by using your own.

One of the interesting challenges to this system is the need to modify existing tables on the fly. I can think of at least three solutions. Build a sync table state myself, use Hibenate's method and do a create temp, copy, remove original and rename (like SQLWB). No comment on that yet.

Security - Role based security is a common theme, but what is needed is actually feature, row and field level security with the additional issue of allowing to specify permissions on role, nested roles and users. Care should be taken to ensure that we can replace that with other security options, such as limiting a user's ability to submit questions to products the user have previously bought.

Authentication - we probably want mixed mode authentication, with the option to define users that are not in Active Directory.

Full text indexing of everything - just something to toy with as well, push luecene for everything there, there should be no reason that I couldn't search for customer:3424 and get it.

I got some nifty ideas about how to implement the security infrastructure, but it is 05:21 AM already, and I am not sure why I am still awake.

Let me know what you think.

Comments

Mark
11/17/2007 04:34 AM by
Mark

I think you are right on target. And so does Scott Hanselman based on this document. http://www.hanselman.com/blog/content/binary/Hanselman%20-%20Managing%20System%20Deployment%20with%20PowerShell.pdf

Its not just about powershell but also how the entire deployment environment is under change control using subversion and where customers managed and deploy things themselves.

Have you read it?I

Marco
11/17/2007 08:29 AM by
Marco

Nice overview! Exactly what you need in many applications.. When shall we start building ? ;-)))

Robert M.
11/17/2007 11:16 AM by
Robert M.

Nice article Ayende ... it is extremely interesting for me...

At the moment I am more interested in the security infrastructure that you mentioned at the end of the post. I've already built a kind of framework for user management, shared user profile and single sign-on for our internal and external applications and users using plenty authentication methods like u/p, AD, RADIUS, cryptocards, SIM challenge. Apart from basic authorization based on roles, I am planning to go further and support also advanced features like: assignable permissions, hierarchical securable resources and a combination between role-based security and permission based security.

Everything is centrally administered and tasks can be delegated to lower rights power users. Disabling a user, for example, has immediate effect and the user automatically is prevented to login in any integrated applications where user had the right to access it.

One of the must haves that you forgot to mention about the dream enterprise platform is the support interop and integration with existing platforms and services already in the target company. No enterprise platform, no matter how well designed and extensible is built, will exist in isolation. Large companies like the one for which I currently work use myriad of systems and technologies in running their business. Invoices and customers are considered master in the billing platform, products and services are defined in the ERP/financial system, customer interaction and events are kept in the CRM platform. Retail system, for example, sucks data from all of these platforms providing an unified environment for sales representatives. Considering all the above, not providing interop services (the lingua franca between UNIX/C, .NET/CLR, UNIX/Java) is considered as bad design and will not ever be accepted for production use.

Therefore, my .NET framework has APIs for multiple platforms and can be accessed from Java, PHP, Delphi, .NET, C++ apps not matter the platform on which the member system is runing: Win32, CLR, JVM, Unix ELF etc.

Could you please detail a little bit more on how do you envision the framework for data security? For example, how would you build a system which should show to managers an aggregated view of all their subordinates transactions and how team leaders will only view a vertical slice of data depending on the permissions? Where would you store metadata about these security levels?

Regards,

Robert

Thomas Krause
11/17/2007 12:18 PM by
Thomas Krause

Sounds good to me so far...

How exactly would upgrades work? Assuming you ship a core set of entities with your application and in your next version you have some small changes in the controller for an core entity. The customer also changed something in the same controller. Will the SCM server simply perform a merge operation for these updates? Or would these core entities be simply 'read only' to avoid any update problems, but of course limiting the ability to change the system.

Also what about versioning of the database itself?

Steve Campbell
11/17/2007 02:53 PM by
Steve Campbell

I guess everyone focuses on what is most familiar to them. For myself, the workflow part is interesting. I have used the proposed style before, although I have called them "transaction enlistees".

For full functionality, the three things you need with them are some sort of unit-of-work (so they can participate in a transaction), a way to manage them (perhaps via the controllers), so that they are only "registered" when they need to be, and finally a way to tell what has changed (maybe your unit-of-work will do that for you).

An unrelated (to workflows) warning: Scalability is not free.

Bart Reyserhove
11/19/2007 08:28 AM by
Bart Reyserhove

Ayende,

I read the following in your post:

"This screen will be responsible for generating three files. /entities/Account.boo, /controllers/AccountController.boo, /views/Account/edit.brail

These files represent the way the sum total of information that the system needs to know about an entity. Those are also versioned files, and more importantly, they are very malleable for user modifications."

Does this mean that there is no recompilation necessary?

Ayende Rahien
11/19/2007 05:01 PM by
Ayende Rahien

Bart, precisely. The next part is going to talk about hot deployment, which is part of this.

Comments have been closed on this topic.