Ayende @ Rahien

It's a girl

Macto: Looking at warrants

After spending so much time talking about how important Warrants are, it is actually surprising to see the UI for a warrant:

image

It is pretty simple, because from a data entry perspective, there isn’t really much to it. It is the effects of the Warrants that make it such an interesting concept. One thing to note here is that the date that we care about for the Warrant isn’t the date it was issued, but when the guy was actually arrested, that is where the clock starts ticking.

Adding a Warrant is also not that complex from a data entry perspective:

image

image

image

As you can see, thus far, it is pretty simple. But when you click the Finish button the complexity starts.

We need to check that the Warrant is valid (issued by someone with the authority to do so), and then calculate the new duration for the Inmate.

And that is enough for today, I just wrote ~10 posts on Macto in the last 24 hours, it is time to do something else.

Tags:

Published at

Originally posted at

Comments (6)

Macto: Talking to nasty people

Well, so far we have looked at the main screen and the Counting screen, it is time we introduce ourselves to the Inmate screen as well:

image

As you can see, there isn’t a lot going on here. We have the Inmate’s Id, name and location (including noting where he is located, if he is known to be outside the prison). We don’t usually care for things like detailed information in the normal course of things (the Inamte’s national id, for example). That information is important, but usually not relevant, we relegate it to a separate screen.

The dates for incarceration and scheduled release are also important, but they aren’t available for editing, they are there only for information purposes.

The note is there to make sure that highly important information (such as whatever the Inmate is suicidal / flight risk) would be clearly available. The same is true for the Cliff Notes version of the Record.

It is there not for casual use, but to ensure that pertinent information is available. Most importantly, note the last line there. Which means that if this Inmate is about to be released, we have to notify someone and get their approval for that. Well, approval is a strong word, we notify them that they need to give us a new Warrant for the Inmate, but we can delay releasing him (say, until midnight the day he is to be released) waiting for that Warrant.

Warrants are important, and you can see the ones that this guy have listed here. The last Warrant is the important one, the others are shown for completion sake and to ensure that we have continuous Warrants.

There are also several actions that we can do to an Inmate. We can Transfer him to another prison, Release him or Add a Warrant. Each of those is a complex process of its own, and I’ll discuss them later on.

Tags:

Published at

Originally posted at

Comments (7)

Macto: Counting is The Holy Grail

I might have mentioned before that Counting is somewhat important in prison. That is about as accurate as saying that you would somewhat prefer it to keep breathing. Counting is the heartbeat of the prison, the thing that the entire operation revolves around.

Here we can see the counting screen:

image

You can see that we have an Open Count here, that is a count that is still in progress. Some cell blocks have reported their counts, and some are still in the process of making the count.

A Count is Closed when we know where are the Inmates are. (When the two numbers line up properly). You probably noticed that there are two ways to report a count, the first is for Inmates who are outside the prison (court, hospital, etc). Because they are outside the prison, we track them by name. For internal counts, we don’t really care about names, just the numbers.

There is another type of a Count, a Named Count, which is a process that happen very rarely, but is usually used to reconcile what is in the computer and what is actually in the cells.

It is important to understand the “Officer in charge” field, basically, it is the guy who has the legal responsibility and is signing off on those numbers. In other words, if there is something wrong, that guy is going to take a hard fall.

Tags:

Published at

Originally posted at

Comments (14)

Macto: Getting Started, you never forget your first Inmate

People always love to start with CRUD, but it is almost never that simple. In this post, we will review the process required to accept an Inmate into the prison.

That process is composed of the following parts:

  1. Identification
    • Who is the guy?
    • Id number
    • Names
    • Photo
    • If we can’t identify / refuse to identify, we still need to be able to accept him.
  2. Lawful chain of incarceration
    • Go over all the documents for his arrest
    • Ensure that they are all in order
    • Ensure that they are continuous and valid
    • Check if there are any urgent things to do with him. For example, he may need to be at court today or the next day.
  3. Medical exam
    • Is it okay to hold the guy in prison?
    • If he is not healthy, can we take care of him in prison?
    • Does he require hospitalization?
    • Does he require medicine / treatment?
    • Are there any medical consideration into where to put him?
  4. Intelligence
    • Interviewing the guy
    • Report interesting details that are known about him
  5. Acceptability
    • Does he fit the level of Inmates we can accept? We usually don’t put murderers in minimum security prisons, for example.
    • Does he have any medical reason to reject him?
    • Are there any problems with the incarceration documents?
    • Is there any intelligence warning about the guy?
  6. Placement
    • Decide where to put the Inmate
    • What type of an Inmate is he? (Just arrested, sentenced, sentenced for a long period, etc)
    • Why is he in prison for?
    • What kind is he? (You want to avoid Inmate infighting, it creates paperwork, so you avoid putting them in conflict if possible)
    • Where there is room available?

Another important aspect to remember is that while we are allowed to reject invalid input (for example, we are allowed to say that the id number has to consist of only numeric characters), we are not allowed to reject input that is wrong.

What do I mean by that. Let us say that we have an Inmate at the door, and he doesn’t have his incarceration paperwork in order (well, not he, whoever brought him in, but you get the point). That means that legally, we can’t hold him. But Macto isn’t where things are actually happening, it is merely a support system that tracks what is going on in the real world. And the prison commander can decide to accept that guy anyway (say, because the paperwork in en route), and we have to allow for that. If we try to stop people from doing this, it is going to be worked around, and we don’t want that. The system is allowed, even encouraged, to warn the users when they are doing something wrong, but it cannot block it.

The first part, Identification, is actually pretty easy, all told. This is fairly simple data entry process. We’ll want to do some checkups on the data, such as that the id number is valid or to check the id against the name, etc. But we basically have to have some level of trust in the documents that we have. You usually don’t have an arrest warrant for “tall guy in brown shirt”. If we find any problems there, we can Flag the Dossier as a potentially fraudulent name. This is also the stage where we want to check if the Inmate is a returned visit, and bring the life the old Dossier.

The second part is more complex, because there are many different types of Warrants, each with their own implications. Arrest Warrant is valid for 24 hours, Remand Warrant is good until sentencing, etc. We need to input all of those Warrants, ensure that they are consistent, valid and continuous. If there is a problem with that, we need to Flag the Dossier, but we can’t reject it. We will discuss this in more detail in the next post.

The third part is basically external to Macto, we may need to provide the Inmate ID, for correlation purposes, but nothing beyond that. We do need to get approval from the doctor that the Inmate is in an OK condition to be held in the prison. That does get recorded in Macto.

The forth part is again, external to us. Usually any information is classified and wouldn’t appear in Macto. We may get some intelligence brief about the guy, but usually we won’t.

The fifth part is important, this is where we actually take legal and physical ownership for the Inmate. Up until that point, we had him in our hands, but we weren’t responsible for him. Accepting the Inmate is a simple matter if everything is good, but if the Dossier was Flagged, we might need approval from the officer in charge. Accepting an Inmate means that he is added to the prison’s Roster.

The sixth part is pretty much “where do I have a spare bed”, after which he is added to the Roster of the cell block he is now in care of.

It is important to note that Placement always happens. Even if immediately after Accepting an Inmate you rushed him to the hospital, that Inmate still has to be assigned to a cell block, because that assignment means that the cell block commander is in charge of him. That way we avoid potential mishaps when an Inmate is assigned to no one, doesn’t get Counted.

Okay, I think that this is enough for now, in the next post, we will discuss what exactly goes on in the second part, it is a pretty complex piece, and it deserve its own post.

Tags:

Published at

Originally posted at

Comments (13)

Macto: The Main Screen

I usually like to think about the responsibilities of the system by showing up the UI. It is a great way to communicate with both customers and developers.

Here is the main screen for the application:

image

This is actually a bad place to start with, in terms of coding start points, because it requires so many other things as well. This particular screen is likely to be viewed pretty much everywhere, this is what the prison commanders and the cell blocks commanders have at their desktop, what the officers are using to do their daily work (usually for Counting, admittedly).

Even before we can start , it reveals quite a lot about the actual way things work. In this screen, we can see that we have Flagged Dossiers, those are Inmates that have some problem with their Dossier. We can accept Inmates with problematic Dossiers, but we want to fix that as soon as possible, so we make this something that is very much front and center.

The “Action required” section detail Inmates that we have to take some action about. Whatever it is a court date that this inmate have to be at or his sentence is ending or a warrant that need extending.

Finally, and most importantly, we have the counts, which are the most important thing in the prison. You can see that the numbers at the bottom line up. If they don’t, we have A Problem.

Tags:

Published at

Originally posted at

Comments (16)

Macto: Warrants are for fools

Warrants are kinda important in a prison. They are the legal authority to limit someone’s freedom. In wouldn’t be overstating the fact in saying that Warrants are one of the major factors being managed in Macto.

There are all kind of Warrants in existence. To list just a few of them:

  • Arrest Warrant – Issued by an officer, generally hold for 24 hours only.
  • Detention Warrant – Issued by the court, generally for a short amount of time, up to a few weeks, in most cases.
  • Remand Warrant – Issued by the court, generally instructing the prison to hold the Inmate in custody until sentencing (not limited in time).
  • Sentencing Warrant – Issued by the court, specifying the total time that an Inmate is to be incarcerated.

There are other warrants, such as an Court Arrest Warrant, for example, but for the purpose of Macto, we won’t get into those. The type of activity currently required by the prison doesn’t really need them, but that might change in the future.

There is also another type of Warrant available, it is Whatever The Judge Said Warrant, or as the lawyers call is Mandamus Warrant. It is basically an instruction to do something, and it can be just about anything. From letting the Inmate to call his wife to putting him in a different cell or transferring him to a different prison to command special food / treatment to… Well, there is a reason I call it Whatever The Judge Said.

The rules for Warrants for incarceration are pretty simple. Each warrant type has an issuer (Arrest Warrants can only be given by Officers of rank Captain and above) for a certain duration (which in some cases, may be limited, such as the 24 hour limit for Arrest Warrants). Depending on the type of Warrant, it can be in Hours, Days, Months or Years. The units in which a warrant is specified are very important. In particular, there is a difference between 30 days incarceration and 1 month incarceration, for example. And hourly Warrants requires that by the time the Warrant expire, you are either got a new one at court or let the Inmate go.

The last issued Warrant is always the one that is valid, and all Warrants must be continuous. Gaps in the middle are considered to be a Very Bad thing.

Tags:

Published at

Originally posted at

Comments (27)

Macto: Non functional concerns, you are a legal system

Macto is a system that operates in a highly legislative environment. As such, we have to be prepared for the court to ask us to show our records about a particular Inmate. Part of that is ensuring that we preserve the history of the Inmate’s Dossier. An example where this would be relevant is when a lawyer contend the legality of incarcerating the Inmate. You have to show not only that you have legal authority to incarcerate the guy, you also have to show that you had that authority continuously throughout the incarceration period.

A typical case where there is a problem is shown below:

  1. 27 June 2011 20:52 – Arrest by Sargent Azulay for car vandalizing.
  2. 29 June 2011 09:15 – Detention, 8 days by Judge Judy
  3. 5   July 2011 – Remanded in Custody by Judge Thachil Oti
  4. 14 Aug 2011 – Sentenced, 3 months by Judge Koev Li
  5. 27 Sep 2011 – Released at end of sentence

Do you see the problem? You probably don’t, but for me, it shouts. The issue is that an Arrest is only valid for 24 hours. Because of the gap in the incarceration warrants, a lawyer can usually get an Inmate out.

That means that part of what the system has to do is to be able to say not only what the current state, but what was the state at any given point in time. Those are usually called Temporal Systems, or Append Only systems, since you are not allowed to make modifications existing data, only create new data.

They also tend to be quite hard to work with, but this is still isn’t a post about the technical stuff, so we will let it go until we get to the good parts.

Tags:

Published at

Originally posted at

Comments (7)

Macto: And it goes on your permanent record, too!

According to the way the blog posts are currently scheduled, I just spent about a month doing nothing but talking about stuff that has very little to do with the implementation, code or even just rough architecture. I bet you thought that you were going to see some code, diagrams and something real that you can sink your teeth into…

Well, not so fast, this is supposed to be a DDD sample, as such, the first and foremost topic that discuss is the actual domain. I think that this is likely to be the last of the pure domain posts, and I’ll get started with the actual design stuff shortly. But before we do that, we need to learn about one last aspect of the domain, the Inmate’s Record.

So far, we have dealt mostly with the Dossier, the legal stuff that means that we can keep an Inmate in lawful incarceration, but in addition to that, we also have the Inmate’s Record. The Record is basically all the interesting things that the prison stuff needs to know about the Inmate. Those things range from cliff notes version that you need to look at before you interact with an Inmate to a detailed record of his stay in prison.

The cliff note version is usually used in briefing about the guy, “Look, we have to take him to his court date, you need to remember, the guy is on suicide watch, so never leave him alone…”. In the cliff note version, we highly important aspects of the Record. Suicidal, Flighty (tried / planning to escape), Avoid Putting With Inmate X, etc.

The Full Record is used for things like intelligence reviews, interviews, parole hearings, and in any case where there is a need to learn a lot about the Inmate.

What goes into the Record?

  • Guards’ reports
  • Intelligence gathered
  • Disciplinary actions

And probably a whole lot more that I am forgetting. It is important to note the difference between the Dossier, which is usually handled by Legal and the Record, which is usually handled by Staff. They both refer to the same Inmate, but they are usually handled, maintained and used completely separately.

Tags:

Published at

Originally posted at

Comments (6)

Macto: Once more from the top, I swear I had a few more over there

Counting is more than just a regular event in prison, it is more like the heart beat of the entire operation. Indeed, one of the more disruptive events in a prison is when Inmates refuse to be Counted. That is ranked up there will a full scale riot.

Macto is meant to be mostly about the legal aspects of an Inmate’s incarceration, but it can’t ignore the Counting. Indeed, we need to explicitly support those. Just to make our life complicated, when Inmates are Counted, they don’t actually have to be Counted (except at Opening Count and Closing Count, of course), they just have to be Accounted For.

For example, an Inmate may be at the Courthouse during Noon Counting, and that is just fine, as long as we know that he is there. Or there might be an Inmate that is present in another cell during the day, which is also pretty common.

Any changes for the counting for Closing Count are usually pretty extraordinary, something being hospitalized, just arriving from a very long day at Court, etc.

In Macto, we need to record not only that the counting has been made, but also:

  • How many Inmates where present?
  • How many Inmates where supposed to be there?
  • If there are any discrepancies, are they accounted for?

Oh, and you can’t just not accept invalid data, because if an Inmate has Escaped, you still need to be Count all of the rest (in fact, you want to be able to Count them very quickly, nothing make sure that you’ll Count as a suspected or real Escape attempt).

Speaking of which, there is another aspect of Inmate management that we haven’t spoken about yet, the actual tracking of the inmate, but I’ll discuss that in my next post.

Tags:

Published at

Originally posted at

Macto: Day to day life

So the officer shows up in the morning, logs into Macto, and… what does he sees? What are the day to day operations that are required?

This is usually much harder to figure out, because there isn’t any particular action that initiate things, it is usually the routine stuff that trips you.

Since we are mostly interested in the Inmates legal statuses, every day, we need to start with an Action Plan:

  • Which Inmates go home today?
  • Which Inmates’ incarceration should be extended?
  • Which of the Inmates need to go to court?
  • Notify interested parties about Inmates who are scheduled to be released soon.
  • Are there any Inmates who should have been freed but are still hanging around?

Again, note how limited our scope is. We don’t deal with things like cell searches, scheduled drills, etc. Those are happening in any reasonable prison, and they probably need to be tracked, reported on, and scheduled. But those things are pretty much Routine Military Activity (same as the requirement that every soldier re-qualify on firearms once in some period), and there is probably software out there that already does it. We are focusing on the Dossiers, and that is a complex enough world on its own.

Did you notice the actual difference between the first two items of the action plan? What is the difference between them?

Inmates which gets to go home are usually those that were sentenced and served their time. Inmates whose incarceration should be extended are Inmates whose authority for incarceration is time limited, and would have to be released. However, there is usually a reason why they are incarcerated, and that usually means that instead of letting them go, we have to take them in front of a judge to extend the incarceration period until they are finally sentenced.

This part of the system is basically reports and alerts. It gives the user the information about what sort of actions should be taken to ensure that we don’t run into habeas the curpus scenarios without good answers.

Tags:

Published at

Originally posted at

Comments (4)

Macto: Where is the Inmate anyway?

Accepting a new Inmate into prison is usually composed of the bureaucracy in the beginning, and ends with the Inmate arriving at his bunk. The last part is actually pretty complex.

Deciding where the Inmate would go is a decision that is composed of many factor:

  • What type of an Inmate is he? (Just arrested, sentenced, sentenced for a long period, etc)
  • Why is he in prison for?
  • What kind is he? (You want to avoid Inmate infighting, it creates paperwork, so you avoid putting them in conflict if possible)
  • Where there is room available?

I am skipping on other stuff, but I think that you get the picture.

The Inmate’s location is another thing that seems simple on the surface but gets complicated when you drill down. The Inmate’s location is actually compromised from several different aspects. First, and the most obvious one, is the actual physical location of the Inmate. For example:

  • Cell Block B, Section D, Cell 349
  • Redbrick Hospital, ER
  • Vacation
  • East Misphat Courthouse
  • “Loaned” to another prison

All of those are pretty obvious (loaning an Inmate is rarely done, but can happen if he has to do something like show up at a court near the other prison that is far from his current prison).

The next aspect is his location is who is signed for this prisoner. That is a problematic concept if you don’t understand how prisons work. You can think about it as a chain of responsibility. Since the Inmate is in Lawful Custody, if something happens to him, then someone is going to answer some questions. We used to have a joke, “you break, you replace”.  Who signed on for this Inmate is basically who has the legal responsibility for this Inmate. This can be:

  • Cpt. Yom Kashe, Commander Cell Block D
  • Lt. Halach Alley, Escorting to Courthouse
  • Sarge. Yashnoni, Guarding at Hospital

I think that you get the picture. And finally, we have who has overall responsibility for this Inmate? Put simply, while the Inmate is hospitalized for a week, it may be Sargent Yashnoni who is actually standing over the bed, but it is Captain Kashe that has the responsibility for the guy. He is the one who has to report him as “not present, location is known”.

Tags:

Published at

Originally posted at

Comments (5)

Macto: Let’s CREATE an Inmate

After spending so much time talking about the application, I think that you can understand why this isn’t going to work:

image

For a start, we don’t create an Inmate. A prison just doesn’t have this concept.  Second, the process of accepting a new Inmate into the prison is a pretty complex one. An Inmate can come from one of several sources. It can be a guy just brought in from the streets or it can be an Inmate from another prison (and then it depends on where he came from), etc.

Each scenario requires different workflow for processing the new Inmate. And what about errors in the process? For example, let us say that your prison is rated for 14 years old and up, and suddenly you have at the gate an officer with three 13 years old that were just arrested with suicide belts. Legally, you aren’t allowed to hold them. Practically, you can’t really let them go. (For the record, that was a real scenario that happened, I had to get a judge to approve this.)

Here is another true story. I sent an Inmate to another prison, and along the way, the escort manage to forget the documents (those are the oh so important that the sky fall down and smacks you in the head if you don’t have them) in my prison. They got to the other prison around 2 AM, and only then discovered what happened. I was able to convince that prison’s commander to hold the guy until morning in his prison until I can send the documents.

Validation? You make me laugh. This is the real world, and if your system reject an Inmate that the prison commander has decided that he is going to accept… well, that Inmate still has to be tracked, but now it is offline and probably done on the back of some envelope somewhere. The system may alert and warn, but it cannot deny people.

Welcome to the real world.

Overall, the process for accepting a new inmate is quite simple, at its core. You need to be able to identify the Inmate, and you need to have a valid chain of lawful incarceration. The first might be a problem (the guy might have been just picked up, have no idea and refuse to identify himself). The second might also be a problem, as we have seen above.

But eventually, we have the Dossier for an Inmate, which contains name, id, picture and the chain of lawful incarceration. The next step is to place the Inmate in the prison, and that is a whole different key of gunpowder. I’ll discuss that in my next post.

Tags:

Published at

Originally posted at

Comments (12)

Macto: Creating The Model

Actually, starting with the model is usually a bad idea, it ties you to a particular representation of the application, usually as codified in the database schema / UML diagrams. Instead, I rather like to talk about responsibilities.

In the case of Macto, we need to figure out what the system is actually doing. It is an incarceration system, and it seems pretty clear that the root of everything in a prison is in the Inmate, right?

Not so fast!

As a matter of fact, the prison care very much about the Inmate, but Macto? Not so much. That is because we need to make a distinction between what happens in the Real World and what actions are required by our system. In the case of Macto, we are actually managing the Inmate’s Dossier. I thought about calling this a File, but I thought it would be a confusing term, considering how often we use this term technically.

Why is the Dossier so interesting? Because from a IT system perspective, it is the only thing that actually happens in a prison that can be captured inside a computer. For example, does it make sense to make a system for managing garbage removal from the cells?

Sure, if we could RFID all of the inmates, that would give us great IT experience (and save a lot of time and effort), but I am going to assume a prison where this isn’t possible, where most of the actual interactions with the inmates is actually happening by real live people. That means that what we actually care about for Macto is a fairly narrow aspect of what it means to manage a prison, the Inmate’s Dossier.

And after talking so long about why we care only about it, lets talk about what it is. The Inmate’s Dossier is the folder in the registration desk where all of the relevant information about a particular inmate is kept. Here is how it looks like:

image

What is stored inside that dossier? Well, all the really important aspects of being an Inmate. First and foremost, we have the chain of lawful authority for incarceration. What is that?

Let us take a very simple case, of a guy caught vandalizing a car. The chain of lawful authority for incarceration would be something like:

  1. 27 June 2011 20:52 – Arrest by Sargent Azulay for car vandalizing.
  2. 28 June 2011 14:52 – Detention, 8 days by Judge Judy
  3. 5   July 2011 – Remanded in Custody by Judge Thachil Oti
  4. 14 Aug 2011 – Sentenced, 3 months by Judge Koev Li
  5. 27 Sep 2011 – Released at end of sentence

What does all of this mean? Well, a lot, actually. At #1, we had a guy that was arrested by the police, we then have 24 hours to bring him in front of a judge, to make a determination about the case (a small matter of Habeas Corpus). At #2, the judge gave us 8 days to complete the investigation while the guy is still incarcerated. At #3, we showed up in front of a judge and convinced him that we have a strong enough case to keep the guy in jail until sentencing. At #4, we got the final sentence, but calculating it takes some thinking, it is a sentence from the time of the incarceration, which means that the guy spent a total of 3 months in jail, not 3 months from sentencing.

The chain of lawful authority for incarceration is incredibly important, because without it, you are Unlawfully Incarcerating and that is a Bad Thing.

Please note that I am drawing off the experience of prisons that I was at, mostly high security military prisons for terrorists. In a civilian prison, you have to worry about additional things, like work release programs, rehabilitation efforts, etc. I am not going to touch those issues in Macto, though.

What other things are going to go into the Dossier, the ability to answer quo warranto inquiries is probably the most important one, but there are other things going on there as well. For example, we track where is the Inmate located (not a trivial matter, actually). We need to track Reports on the Inmate (aggressive, suicidal, etc).

There are probably more than that, but I think that this is enough for now. You may have noticed that so far, I haven’t shown you any UML nor have I started with the database schema. To be perfectly frank, it is not relevant at this stage at all. On the next post, I’ll start talking about the basics of CRUD operations.

Tags:

Published at

Originally posted at

Comments (22)

Macto: The boundaries of a prison

Macto is a prison management system. To be rather more exact, it is an incarceration management system. Managing a prison is a very complex process (did you ever think about who takes the trash out? Or how to handle things like watch rotations?).

I am getting ahead of myself, however. The first thing that we have to do when we start designing a system is check:

  • What is the scope of the system?
  • What is the intended usage?
  • Who are the users?
  • What other systems are we going to interface with?

Those are the basics, because in a lot of projects, they don’t get asked.

Running a prison is a complex task, I already said. If we wanted a single piece of software to do it all, we would have to do a lot of stuff that we don’t really need. For example, are we going to write our own payroll system for the prison’s stuff? Our own medical software to manage the infirmary? What about managing the inventories in the prison (how many prison uniforms and tear gas canisters we have)? What about the maintenance software for the fleet of vehicles required to run even a small prison?  Or how about… but I think that you get my drift.

There are a lot of aspects where it would be significantly better to get an of the shelve solution rather than try to write our own version of that. We need to be focused where we would provide actual utility to our customer.

That is what I said that we are going to build incarceration system, not a prison management system. Macto is all about managing the aspects of a prison that are different than those of any other business.

You can read about Macto in my initial post about it, here, since it put the context of the system in a nice perspective.

I am not sure if there will be code available, but I am going to outline my thinking here, and I think that this would be valuable in itself.

Macto is going to be about managing the aspects of a prison that you can’t get off the shelve. In order to do that in a way that would make sense to you, I need to first explain what we are going to be doing.

Like any good architect, I’ll be doing that by starting with the model, which is also the cue for the next post.

Tags:

Published at

Originally posted at

Comments (5)

Making Macto into an OSS project? Volunteers needed

I don’t have the time to actually sit down and build Macto, but I got a couple of people asking about another option. Making Macto into an OSS project that will build this application under my guidance and assistance.

What do I have planned?

Assuming I manage to get a few people (and I’ll probably limit it to a small number), I’ll open a private mailing list and github repository and we will start working on that. I’ll provide guidance and direction for the project, as well as reviewing everything.

This is going to be a private project for about a month, after which we will hopefully have something at hand that we can talk about and show around.

Any volunteers?

Update: Comments are closed, we have enough volunteers

Tags:

Published at

Originally posted at

Comments (19)

Ayende’s DDD Application: Macto

A long while ago I planned on doing a series of webcasts, demonstrating how to build a real Enterprise System using my approach. I called it Macto, but I never got around to it.

The recent discussion around the Microsoft N Layer sample has caused me to rethink this decision. I think that it would be a good idea to do so. I don’t know if I’ll complete it, and I don’t know if I’ll do code, but I intend to lay out a full architectural approach. I already started writing the series of posts, and it should air out starting around the 18th of July.

Update: Okay, I moved some posts around, it is now starting next week, but there is a lot of additional content, so it is going to be posted over the next month or two.

Macto, Authorization decisions

image

Authorization is one of the few ubiquitous requirements, you are going to have to handle them in pretty much every system that you are going to build.

The users are the staff, and the securables are the inmates. The problem is that we have fairly different authorization requirements for different parts of the system.

For example, any authenticated user can lookup pretty much any inmate’s data (except for medical records, of course), but changing release date is something that only Legal can do. Only the staff on the enclosure that the inmate is located in can Sign Out the inmate. Actually releasing an inmate requires Legal and On Duty Officer approval, etc.

However, during weekends, the on duty staff assume responsibility for the entire prison. That is, an officer from enclosure C can Sign Out an inmate from enclosure A if that officer is the on duty officer.

There are a few more complications, but we will ignore them for now. One thing that is fairly clear, we have fairly complex authorization requirements, and they are different for each part of the system. For that matter, the way we make security decisions itself is different.

And since authorization decisions are synchronous (you can make them async, sort of, but at very high cost), performance is a critical concern. This is especially true because there is a strong tendency to call authorization decisions many times.

Given that, and given the complexity inherit to authorization, I think that we can skip the entire problem entirely by changing the rules of the game.

Most authentication systems would have you do something like:

auth.IsAllowed(CurrentUser, "/Inmate/Move", inmate);

And rely on the system to do its magic this way. The problem in this manner is that we provide the authorization system with very little information with which it can work. That means that in order to make authorization decisions, the system has to have a way to access other data (such as in what enclosures the current user is in charge of, where the inmate is, what is its status, etc).

The problem then become more an issue of data retrieval complexity rather than the authorization rules complexity. I think that we can avoid this by designing the system with more flexibility by providing the required data to the authorization system explicitly.

What do I mean? Well, just take a look:

auth.IsAllowed(
	new SignOutAuthorization
	{
		OfficerRank = CurrentUser.Rank,
		OfficerRoles = CurrenUser.GetAllCurrentRoles(),
		OfficerEnclosures = CurrentUser.GetEnclosuresUserIsResponibleFor(),
		InmateEnclosure = inmate.Enclosure,
		InmateStatus = inmate.Status,
	}
);

In other words, we are explicitly providing the authorization system with all the data that it needs for a particular task. That means, in turn, that we can now execute the authorization decision completely locally, without having to go somewhere to fetch the data. It also open the option of using a DSL to build the authorization rules, which will make things more dynamic and easier to work with.

Macto, defining Centralized Service, Distributed Service and Localized Component

I lately come into the conclusion that I need a few new terms to describe a few common ways to talk about the way that I structure different components in my applications.

Those are Centralized Service, Distributed Service and Localized Component. Mostly, I use them as a way to express the distribution semantics of the item in question.

As you can probably guess, I am using the term service to refer to something that we make remote calls to, while I am using the term component to refer to something that is running locally.

Centralized Service is probably the classic example of a web service. It is a server that is running somewhere to which we make remote calls to. As far as the system is built, there is only one such server. It may be implemented with clustering or load balancing, but logically (and quite often, physically), it is a single server that is processing requests. This is probably the easiest model to work with, since it more or less remove the entire question of concurrency conflicts from the system. Internally, the Centralized Service is using transactions or locks to ensure coherency in the face of concurrency.

Distributed Service is a built upfront to run on a set of server, and the need to handle concurrency conflicts is built into the design of the system. That may be done using sharding, Paxos or other methods. Usually, we build Distributed Service for very high scalability / reliability cases, since it tend to be the more complex solution. An example of a Distributed Service would be DNS, where we explicitly design the system to be resilient to failure, but accept the more complex concurrency issues (slow updates).

Localized Component is a solution to the chatty interface problem. There are quite a few scenarios where we need to make calls to a separated subsystem, but the cost of network traffic completely outweigh the cost of actually performing the operation on the other side. In this case, we may switch from a Centralized Service to a Localized Component. What this means is that instead of executing the operation on the other side, we perform it locally.

In practice, this means that we need to design our system in a way that any data that we would like to have is structured in such a way that it can be brought locally or retrieved very cheaply. An example of such a system appears in this post, although that is a fairly complex one. A more common situation is a component that deals with a set of rule, and we simply need to get the rule from the rule repository and execute it locally.

Another alternative for Localized Components is to structure it in such a way that retrieving and persisting the data is cheap, and processing it is done on locally. That way, the sharing of the data and the actual processing of the data are two distinct issues, which can be resolved separately. A common issue that needs to be resolved with Localized Components is consistency, if the component allow writes, how do other instances of the component, running on different machine get notified about it.

I tend to avoid Distributed Services in favor of Centralized Services and Localized Components, which tend to be easier to work with. It is also easier to lean on existing infrastructure then write an implementation from scratch. For example, I am using Rhino DHT (which I consider to be a Distributed Service) to handle a lot of the complexity inherit to one of those.

Macto, a module features spec sheet for authentication

imageI am going to talk about a few ways of trying to organize a project, mostly as a way to lay out the high level requirements for a feature or a module.

I consider filling in one of those to be the act of actually sitting down and thinking about the concrete design of the system. It is not final design, and it is not set in stone, but in general it forces me to think about things in a structured way.

It is not a very hard to do, so let us try to do this for the authentication part of the application. Authentication itself is a fairly simple task. In real corporate environment, I’ll probably need integration with Active Directory, but I think that we can do with simple username & pass in the sample.

Module: Authentication

Tasks: Authenticate users using name/pass combination

Integration: Publish notifications for changes to users

Scaling constraints:

Up to 100,000 users, with several million authentication calls per day.

Physical layout:

Since the system need to handle small amount of users, we have to separate deployment options, Centralized Service* and Localized Component*. Both options are going to be developed, to show both options.

Feature

Description

SLA

authenticate user

Based on name & password
Lock a user if after 5 failed logins

less than 50 ms per authentication request 99.9% of the time, for 100 requests per second per server

create new user

User name, password, email

less than 250 ms per change password request 99.9% of the time, for 10 requests per second globally

change password

 

less than 250 ms per change password request 99.9% of the time, for 10 requests per second globally

reset password

 

less than 250 ms per change password request 99.9% of the time, for 10 requests per second globally

enable / disable user

disable / enable the option to login to the system

less than 250 ms per change password request 99.9% of the time, for 10 requests per second globally

You should note that while I don’t expect to have that many users in the system, or have to handle that load, for the purpose of the sample, I think it would be interesting to see how to deal with such requirements.

The implications of this spec sheet is that the system can handle about 8.5 million authentication requests per day, and about a 3/4 of a million user modifications requests.

There are a few important things to observe about the spec sheet. It is extremely high level, it provide no actual implementation semantics but it does provide a few key data items. First, we know what the expected data size and load are. Second, we know what the SLAs for those are.

* Centralized Service & Localized Component are two topics that I’ll talk about in the future.

Tags:

Published at

Originally posted at

Comments (13)

Macto, or How To Build a Prison

image

The sample application that I am going to build is going to be a prison management application. I am going to take this post as a chance to talk about it a bit, discuss the domain and then I’ll talk about the overall architecture in more details.

The domain of a prison is actually fairly simple, you have an inmate, and the sole requirement is that you would keep him (it tend to be overwhelmingly him, rather than her) in lawful custody.

The term lawful custody has a lot of implications, which are, in more or less their order of importance:

  • The inmate is in custody, that is, he didn’t manage to run away.
  • Custody is lawful, that is, you have legal authorization to keep him in jail. Usually that means an order by a judge, or for the first 24 hours, by a police officer.
  • Lawful custody itself means that you:
    • keep the inmate fed
    • in reasonable conditions (sleeping quarters, sanitation, space)
    • access to medical facilities. Indeed, in most prisons the inmates get better health care, especially for emergencies, than the people living in most big cities.
    • ability to communicate with lawyers and family

The devil, however, is in the details. I am pretty sure that I could sit down and write about 250 pages of high level spec for things that are absolutely required for a system that run a prison and still not get everything right.

In practice, at least in the prisons I served at, we did stuff using paper, some VB6 apps & Access, and in one memorable occasion, an entire set of small prisons where running on what amounted to a full blown application written using Excel macros.

Anyway, what I think that I’ll do is start with a few modules in the system, not try to build a full blown system.

The modules that I‘ll start with would be:

  • Staff – Managing the prison’s staff. This is mostly for authentication & authorization for now.
  • Roster – Managing the roster of the prisoners, recording Countings, etc.
  • Legal – Managing the legal side of the prisoners, ensuring that there are authorizations for all the inmates, court dates, notifications, etc.
  • Escort – Responsible for actually taking the inmates out for court, medical evacs, releasing inmates, etc.

That is enough for now, for that matter, it is a huge workload already, but that is about the only way in which I can actually have a chance to show a big enough system and the interactions between all the parts.

More on Macto

Looking at the responses that I got, I think that there is some basic misunderstanding about the goal of the sample. Some people seems to want this to be a usable product, some even went ahead and specified some… interesting requirements.

Unlike Storefront, I don’t intend to create something that would be a useful component to take and use, for the simple reason that I don’t think that it would allow to show anything really interesting. Any generic component or system has to either make too much assumptions (constrained) or not enough (open ended). I don’t care to have to hand wave things too much.

Given that I am a domain expert on exactly two things, and that I am not going to create Yet Another Bug Tracking Application, I think that it is fairly obvious what I have to build.

Macto is going to be a prison management system.

I am going to use it to demonstrate several topics that I have been dealing with lately, among them the concepts & features architecture and how to build scalable systems.

I’ll let you stew on that and will post more details about Macto tomorrow.

Macto: An end to end sample

It looks like what people would really like to see is an end to end sample of all the things that I have been talking about lately.

The problem with doing that is the scope that we are talking about here, it is pretty big, and there are some interconnected parts that would be hard to look at in isolation. To make things a little bit more interesting, building a “best practice” application is dependent on far too many variables.

Given all of that, I decided that I might as well copy a good idea and try to emulate Rob Conery’s Storefront series of webcasts. What I absolutely refuse to do, however, is to create Yet Another Online Shop.

Hence, I have another forum, dedicated to this specific task, where you can discuss what you want to see.  We need to decide what is the application, what are the requirements, etc.

Please have the conversation in the forum, which would make it easier to track things.

I think that I can promise at least two or three webcasts to come out of it, including the full source code.