Ayende @ Rahien

Properly getting into jail: The almighty document

Mon, 19 Mar 2018 10:00:00 GMT

This post is the conclusion for this series (unless I’ll get some interesting questions). So far, I outlined how to break apart the system, the data flow and data processing inside it, a lot about the internal constraints and the business logic as it is applied. There hasn’t been a lot of code, because I wanted to keep things at the architecture level rather do low level dive.

Early in the series, I got a comment from Rafal that is pretty profound:

There's a general agreement among software creators and their customers that software replaces papers and going paperless is A Good Thing ™. And then after introducing an IT solution everyone starts complaining that the papers were so much better to work with and allowed for much greater flexibility. Especially for order handling workflow, where you could print copies of the order, hand it out to proper people and be sure they have everything they need to do the job. And you could always put some additional info on the papers when there was a need for special handling.

The rigidity of computer systems often means that we have to work around the system in the real world to get things done. In many cases, that actively hurt the people using the system. For example, if I got an inmate that had a specific constraint (danger to himself, isolated from a particular group, etc), I can take a red marker and write in big letters on the file the message, ensuring that everyone that deals with the file is aware of it. If this is not explicitly called out in the design of the system, there is really no good way to do that with a computer system. And that can be a great deterrent for adopting the system and its usage.

What is worse, if you have such a requirement, it will often show up as something like this:

A mandatory, annoying (and never read) message box that isn’t actually useful for the purpose.

One of the rules that we have as system architects is explicitly anticipating and answering this kind of situations and providing something that can do at least as good as plain old paper.

The design on Macto as outlined in this series of posts attempted to do just that. To continue Rafal’s quote:

And your approach is the same idea applied to software design - make a digital piece of paper that almost physically follows the process, is always there and has everything necessary to do the work, then pass it around and just make sure it's not lost somewhere in between. No central registry, no central decision about where the papers go, just do your task and pass the message to the next station.

Doing something in the UI like getting the user the ability to inject some elements is trivial, after all, if the data format can handle it. So you have a way to record the information the user want and display it in a way that make sense to them, without having to know more about UI design than Right Click > Add (Field / Note / Heading / Timer), etc. At the same time, you gain all the benefits of a computerized system (backups, search, recall, etc), the ability to avoid signing things in triplicate, have access to the entire status of the system at once, etc.

This is not a trivial thing to do, but it can result in quite a different to the system and its adoption.

Properly getting into jail: Data processing

Fri, 16 Mar 2018 10:00:00 GMT

In this series of blog posts, I have talked a lot about the way data flows, the notion of always talking to a local server and the difference between each location’s own data and the shared data that comes from the other parts of the system.

Here is how it looks like when modeling things:

To simplify things, we have the notion of the shared database (which is how I typically get data from the rest of the system) and my database. Data gets into the shared database using replication, which is using a gossip protocol, is resilient to network errors and can route around them, etc. The application will only ever write data to its own database, never directly to the shared one. ETL Processes will write the data from the application database to the local copy of the shared database, and from there it will be sent across to the other parties.

In terms of input/output, the process of writing locally to app DB, ETL process to local shared DB, automatic dissemination of data to the rest of the world is quite simple, once you have finished the setup. It means that you don’t really have to think about the way you publish information, but can still do that in such a way that you are not constrained in the development of the application (no shared database blues here, thank you!). However, that only deals with the outgoing side of things, how are we going to handle incoming data?

We need to remember that a core part of the design is that we aren’t just blindly copying data from the shared database. Even though this is trusted, we still need to process the data and reconcile it with what we have in our own database.

A good example of that might be the release inmate workflow that we already discussed. This is initiated by the Registration Office, and it is sent to all the parties in the prison. Let’s see how a particular block is going to handle the processing of such a core scenario.

The actual workflow for releasing an inmate needs to be handled by many parties. From the block’s perspective, this means getting the inmate physically into the release party and handing over responsibility for that inmate. When the workflow document for the inmate release reaches the block’s shared database, we need to start the internal process inside the block to handle that. We can use RavenDB Subscriptions for this purpose. A subscription is a persistent query, and any time a match is found on the subscription query, we’ll get the matching documents and can operate on that.

Here is what the subscription looks like:

Basically, it says “gimme all the release workflows for block B”. The idea of a persistent query is that whenever a new document arrives, if it matches the query, we’ll send it to the process that has this subscription opened. This means that we have a typical latency of a few milliseconds before we process the document in the worker process.

Now, let’s consider what we’ll need to do whenever we get a new release workflow. This can look like this:

I’m obviously skipping stuff here, but you should get the gist (pun intended) of what is going on.

There are a couple of interesting things in here. First, you can see that I’m writing the code here in Python. I could have also used Ruby, node.js, etc.

The idea is that this is an internal ingestion pipeline for a particular workflow. Independent of any other thing that happens in the system. Basically, the idea is to have a Open To Extension, Close To Modification kind of system. Integration with the outside world is done through subscriptions that filter the data that we care about and integration scripts that operate over the stream of data. I’m using a Python script in this manner because it is easy to show how fluid this can be. I could have use a compiled application using C# or Java just as easily. But the idea in this architecture is that it is possible and easy to modify and manage things on the fly.

The subscription workers ingesting the documents from the subscriptions take the data from the shared database, process and validate it and then make the decisions on what should be done further. On any batch of workflow documents for releasing inmates, we’ll alert the sergeant (either way, we need to release the inmate or we need to figure out why the warrant is on us while the inmate is not in our hands).

More complex script may check all the release workflows, in case the block that the Registration Office thinks the inmate is on is out of date, for example. We can also use these scripts to glue in additional players (preparing the release party to take the inmate, scheduling this in advanced, etc), but we might want to do that in our main app instead of in the scripts, to make it more visible what is going on.

The underlying architecture and shape of the application is quite explicit on the notion of data transfer, though, so it might be a good idea to do it in the scripts. A lot of that depends on whatever this is shared functionality, something that is customized per block, etc.

Properly getting into jail: My service name in Janet

Thu, 15 Mar 2018 10:00:00 GMT

For a long time, whenever I was talking to customers about the business domain, I would explicitly avoid using the term “business logic”. Primarily because I never found such things to be logical in any way shape or form. A lot of the business decisions and policies are driven by a host of legacy reasons, “this is how everyone does it” and behaviors that has became accepted and then entrenched.

Take what is supposed to be a pretty simple rule. Given a warrant, when should an inmate be released? On the face of it, that seems like a pretty obvious and straightforward answer, right?

Depending on the type of warrant (can be for 48 hours, 5 days, 30 months, 10 years, life) the answer is quite different. For example, if someone was arrested at 3 PM on Thursday on a 48 hours hold, he must be released on Saturday at 3 PM. But that is actually a problem, because the prison does not release inmates on Saturday. So the release date is moved back (or forward(!), depending on a lot of stuff).

If an inmate is sentenced for life, that might mean that he is expected to die in prison, be released in 10 years, 25 years or be eligible to go on parole in 14 years and be effectively free. I did some research around sentencing rules around the world and I must say that this is confusing and quite sad. Even within a single legal system the amount of complexity, precedent, custom and variance is staggering.

At any rate, we need to figure out something that seems to be quite simple. Given an inmate and the warrants we have on file, what should be the release date. This can be as simple as having a single warrant, or a series of sequential warrants (arrest, held until trial, sentencing, etc). That is simple and pretty obvious. Go to the latest warrant, get the duration from that and then start computing the release date. Here we have another problem, from what date do we start counting? If the inmate has been under arrest for the entire duration, then we start from that point. If the inmate has been free (bail, etc), then we start from the point he got put back into prison. Sometimes the inmate was held for a while (several months, before getting bail, for example), so that will be counted against the sentencing period (or not, depending on a bunch of stuff). In short, being told “you are hereby sentenced to 10 years” can mean several different release dates, even assuming nothing changes.

So this is complex, and hard, and in many cases very much situational dependent. How do you approach handling this?

To start with, this is one of those cases that you can, should and require to get a specification, complete with examples, test suite and samples, etc. It may sound silly, because all we are doing is computing a date, but the implications are… important, especially for the people who are being held.

The specification will have a few straightforward cases, but also a lot of convoluted mess that with luck, you can get a lawyer to decipher, but most likely not. The way to handle that is to recognize the patterns that you know that you can reliably figure out and provide answers to those. If it was up to me, I’ll be producing a long hand report, like so:

Note that this computation sheet is not the final say, instead, the Registration Office officer is going to sign on that, after having validated the dates independently.

For patterns that aren’t so easy to compute, a good way to handle that is to show the information you have and not give any answer, making the officer that will sign up on the correctness of this result do all the work.

As an aside, printing this report, including the computation and how it was arrived is a really good idea because it can be handed to the inmate as well as to their attorney. At this point, presumably they’ll double check the dates as well. This is important since a mistake in releasing an inmate that didn’t happen yet is free. After all, if an inmate is supposed to walk out on 2022 but we computed the sentence until 2025 and it was discovered in 2018, no harm was done (except maybe to someone’s nerves).

The human in the loop model is quite important in this regard, because of the notion of the single responsibility that I previously mentioned. Someone, a person, in actually responsible for the release date computation, and that should probably be a human that isn’t the system developer from a decade ago.

Properly getting into jail: This ain’t what you’re used to

Wed, 14 Mar 2018 10:00:00 GMT

The inmate population in any typical prison is divided according to many and varying rules. It can be the length of expected stay, it can be the level of security required, the kind of facilities required, the crimes committed, etc.

For simplicity, we’ll talk about Block A (minimum security, good behavior, low risk) and Block C (bitter lifers, bad apples, violent inmates, etc) as our two examples. These differences can create very different environments. Things that would never pass muster in block A and routine in block C and vice versa.

A good example would be the acceptance criteria, in order to be accepted to block A you have to meet certain standards (non violent offenders or 8 years inside with no spots on your records or strong recommendation from an officer). In order to be sent to block C you need to be in a particular kind of trouble (violent crime, recent behavioral issues, high risk from intelligence, etc).

Being written up by a guard in block A will result in loss of privileges like not being sent to work, reduction in visitation, etc. You don’t get written up in block C, you get sent to disciplinary action with the block’s officer and can be confined to the cell, isolation, lose cafeteria privileges, etc.

From the outside, both of these blocks are roughly the same, but from the inside, they have very different populations, behavior and accepted practices.

This means that when we need to write a system that would serve both blocks (and there is also Block B, Isolation and the Medical Ward as well, all slightly different) we are in somewhat of a pickle. How do we account for all of these differences. One way to handle that would be to just deal with the common stuff (the counts, the legal dealing, etc) and let each block dictate policy outside of the system. We can also provide some “note keeping” functionality, such as the ability to assign tasks, keep notes and records on inmates and hope that the blocks would use that so we’ll at least have a record of all these policy decisions.

Alternatively, we can map what each block wants to do and customize the application for each of them. The problem here is that this things change, and when talking about a large enough basis, they change quite often. Given a typical tenure of a block’s officer of about 3 – 5 years (really depend on the type of prison, in some cases, you’ll have tenures as short as a year or two) and the tendency of each new officer to want to make some changes (otherwise, why are they there?) and the fact that in a typical prison you’ll have 3 – 6 blocks and about 10 high level officers that each want to leave their mark (each with independent tenures), you end up with a fairly constant rate of low level changes.

If this make you cringe at the expected number of changes that will be required to always adapt the system, I hear you, that isn’t a fun place to be in.

There are typically two major ways to handle this. Either you’ll ensure that no such changes are accepted, by not making the changes and having the prison work around the different practices while still using the system or you plan to adapt things from the get go. The first option is very common in a top down organization, where the HQ wants to “lay down the law” about how things “should be done”. The other option is typically more expensive, sometimes ridiculously so, depending on how far you want to push it.

Dynamic data, forms and behaviors, oh my! Let the prison guard completely re-design the system in his free time. To be fair, I was a prison guard and I would enjoy that, but I haven’t found many people in my current career that can say that they have prison experience (from either side of the bars). In practical terms, I would say that the technical level of prison guards is at or below the population norm and not at a level sufficient to actually do anything mission critical such as dealing with people’s freedom.

It is actually usually quite easy to convince the HQ people to avoid any flexibility in the system. They like ensuring that things are done “right”, even if that is quite different from how things are actually working (or even can possibly work). But we’ll avoid such power plays. Instead, let’s talk about how we can limit the scope of the work that is required and still gain enough flexibility for most things.

With RavenDB, defining dynamic data is both easy and obvious, so that is easy. Each block can define additional fields that they can tack onto documents and just have them there. The auto indexing features will also ensure that searches on such fields are fast and efficient. I’m not going to touch on any UI elements, that is someone else’s job .

Let us talk about policy decisions. For example, we might need to decide whatever an inmate is acceptable or not for a block. That means that we need to have some way to decide policy. Now, I have literally written a book about building DSLs for just such a scenario. You can very quickly build something simple an elegant that would give the user the chance to define their own policy and behavior.

In fact, given the target audience, this is not a good idea. We don’t expect the prison guard to make such decisions, so we don’t need to cater to them. Instead, we’ll cater to developers, probably the same developers who are in charge of actually building and maintaining the system. This give us a very different flavor to deal with. For example, instead of building a DSL, we can just use a programming language.

For example, we can use JavaScript to shell out at critical parts of the pipeline. A good example would be at the validation stage of processing an incoming inmate. We’ll pass the inmate document to a JavaScript function and that can emit validation warnings and actions that are supposed to take place. Here is a small sample:

The real world would probably have several pages of various business logic around what should and shouldn’t happen here. Including things like assigning to specific cell because of the inmate’s affiliation, etc. The idea here is that we’ll give the developers an easy way to go and modify the behavior of the system for each location this is deployed in.

As an aside, this kind of things needs to be logged and audited. That means that you can store these scripts in something like a git repository and record the commit hash for the version you are using when you are making decisions. In 99.9% of the cases it will not matter, but if you’ll need to show to court why the “computer told us” that a certain inmate had to be dealt with in a certain way, you want to be able to know what happened and produce the right script that help made that decision.

You might also note from the script that the output of the function is a set of warnings, not errors or exceptions. Why is that? Because there is an explicit place here for the human element. That means that if we have warnings for an inmate, we can still actually accept the inmate, despite the warnings. We might require the sergeant to note why the inmate is accepted despite the warnings (and answers may be things such as “they run out of room in B” and “he was overheard saying he would stab someone”). This is because quite explicitly, we don’t treat the system as the source of truth.

This system is the system of record, it holds the information about what is going on, but it isn’t meant to be rigid, it has to be flexible, because we are dealing with people and there is no way that we can cover all situations. So we try to ensure that there is a place for the human element throughout the system design.

Properly getting into jail: Didn’t we already see this warrant?

Mon, 12 Mar 2018 10:00:00 GMT

An interesting problem in distributed work shows up quite frequently in the prison space. Duplicated, delayed and repeated warrants (um… I mean packets). This can lead to some issues, especially since the receiving parties may or may not be in communication with one another.

Consider the case of a warrant to release a particular inmate. It is quite common for the process of actually getting the warrant back to the prison from the court to take a few hours. In that time frame, the inmate’s lawyer has already arrived at the gates of the prison and handed the release warrant to the block’s sergeant (it’s a bit more complex than that, but I’m skipping details here because I want to talk about the technical details of building a system to run a prison rather than the actual details of running a prison).

At this point, the block’s sergeant can pass the warrant off to the Registration Office, which will also accept the warrant from the court at some time in the future or they can initiate the release process directly. What actually happens depends on a lot of factors, but let’s say that they start the release process directly. We already talked about what that would mean, so let’s focus on another aspect of that. How do we deal with the arrival of the warrant to the Registration Office when there is already an open workflow to release the inmate.

For fun, here is an brief example of a warrant:

Yes, this is faxed, often unreadable and sometimes coffee stained. There is no such thing as a warrant id that you can use to check if the warrant has already been seen. There is supposed to be, at least per court / judge, but there often just isn’t.

Side note regarding the issue of faxing warrants. Yes, there have been cases where people just sent a release warrant and people got out. Part of the process of actually processing a release warrant is talking with the court to validate it, but that isn’t always done.

Another fun fact is that what one warrant may do another warrant may undo. So you may have a release warrant on hand, but the court has already issued a stay of 48 hours for that warrant so the police can appeal that, for example. If the second warrant doesn’t arrive in time…

At any rate, the fact that a warrant may show up multiple times and that there may be conflicting warrants being processed at the same time means that there is the need to handle some level of de-duplication. We can usually detect that using the inmate’s details and the date in which the warrant was issued (it is rare that multiple warrants for the same person are issued at the same date, so that is enough to flag things). If the result of two warrants on the same date is the same, we can assume that they are the same.

If there are conflicts, this will raise a flag and require a human involvement to resolve it. A conflict will be raised for any non identical warrants for the same day for the same inmate, because any such activity is suspicious and require additional attention.

Following the Single Responsibility Principle as applied to prison (there must be a single responsible party so we can put them in jail if they mess this up), the validation of warrants is at the hands of the Registration Office and they take care of handling all such warrants. Even if the warrant was served directly to the block’s sergeant, the final validation (and responsibility) is on the Registration Office personal actually signing on the release form.

Properly getting into jail: The workflow of getting an inmate released

Fri, 09 Mar 2018 10:00:00 GMT

The whole point of having a prison is ensuring that the inmates are staying in. As such, the process of actually getting an inmate out of the prison is quite involved. I’m using the release workflow as an obvious example, but there are many such workflows inside the prison that are similar. The inmate’s intake workflow, or disciplinary workflows or even just inmate’s transfers which can be quite complex (and typically also happen in big batches).

In the previous post in this series, I talked briefly about the idea of workflows as an event publication with multiple signatories to the workflow operation. If this make no sense, let me try to explain.

The trigger for starting an inmate’s release can be the acceptance of a warrant for immediate release, the expiration of the warrant to hold the inmate. There are several other variants (transfer to another facility, death of inmate, escape of inmate, etc) that are related but are not too relevant for the discussion, so I’ll just talk about the immediate release and the warrant expiration.

For warrant expiration, the workflow starts from the Registration Office publishing the list of inmates that are supposed to be released today. In technical terms, they create a series of workflow documents that are in the Open state and this gets disseminated to the rest of the system. Immediate release warrant is usually served to the Registration Office, but may also be served directly to the block’s sergeant.

If the Registration Office got the immediate release warrant, things are identical as the usual scheduled release (except that this might be served at any time). However, if this is served to the block’s sergeant, things are more interesting. At this point, it is the block that will initiate the workflow, but the overall responsibility for verifying the warrant and ensuring the actual release is still on the Registration Office.

As an aside, the notion of "who owns this” is quite important in the prison. Mostly because if you mess up, there might be consequences. This can be holding an inmate past the due date (bad, can result in damages paid to the inmate and career consequences to the person who messed up) to releasing an inmate that wasn’t supposed to be released (really bad, sometimes it is not possible / feasible to hold them again, requires court approval, may end up in jail time for the person who messed up and also result in possible dangerous inmate being released). So the idea is that at any time, there is an owner for this process and a clear finger pointing at that person, “You’re to blame”.

Because of this, there are typically multiple steps in the release process. Consider the simple scenario of a scheduled release, we have:

Registering the inmate on the “to be released” list.
Notifying external and internal parties about this inmate’s pending release. For example, an inmate might have to go through a parole officer before actual release. This is done by sending the “to be released” list several days in advanced and getting at least an implicit agreement (by not vetoing / modifying the process) by external parties.
Verification that there are no explicit holds on the inmate. In the case of an inmate that is supposed to be deported, the inmate’s file will have an explicit “deport on release” which typically require coordinating with the border police to handle that. So the inmate can’t just be shoved out the door but handed off to someone.
Identification of the inmate at the block level. This is typically done on the sergeant’s level and then by an officer (preferably from the same block) that are familiar with the inmate and can validate that this is indeed the one to be released.
Checking out the inmate from the block level to the prison’s level. This explicitly remove the inmate from the block’s responsibility once the inmate has been handled off to the release queue.
Identification of the inmate by the Registration Office’s officer. This is a second verification that is done to ensure that there hasn’t been a mix up again.
Verification of the warrant to release the inmate and that there are no newer warrants that are in effect.
Returning of personal affects and getting the inmate’s signature that everything was properly returned.
Checking the inmate out of the prison, this step explicitly ends the period in which the now ex-inmate is held in prison.
Actually getting the newly released ex-inmate out of the prison. This can be to a family member at the gate or to a bus to the nearest city, etc.

I’m probably forgetting a few details in the middle, and there are branches for each and every one of these steps.

In terms of the technical details of how this works. The workflow document is being distributed throughout the system, and then various parties are now in charge of actually completing the various tasks in the workflow. For example, this may mean checking out the inmate’s personal affects from the safe in the morning, preparing for the release. So the steps aren’t necessarily in sequence or ordered.

The important thing is that the workflow document is published, and is now tracked. Typically, the release process is tracked by the Registration Office and the Command & Control Center. The Command & Control Center will typically be involved early in the process if an inmate isn’t released by the typical time, and the close of day process for the Registration Office includes verification that there aren’t any still pending inmate releases.

At any given point, we can track the process of the workflow from any point in the system (remember, changes are made locally and then distributed to the rest of the system). In there is a communication issue between different parts of the prison, that will typically show up as an alert that a particular workflow hasn’t been properly completed in the allotted time. At this point, external channel (walking & talking, usually) is used to verify the status of this particular inmate release.

Properly getting into jail: Services with data sharing instead of RPC or messaging

Thu, 08 Mar 2018 10:00:00 GMT

The design of Macto (that is this prison management application I’m discussing) so far seems pretty strange. Some of the constraints are common, the desire for resiliency and being able to have each independent portion of the system work in isolation. However, the way we go about handling this is strange.

I looked up the definition of a micro services architecture and I got to Martin Fowler, saying:

..the microservice architectural style [1] is an approach to developing a single application as a suite of small services, each running in its own process and communicating with lightweight mechanisms, often an HTTP resource API.

This wouldn’t work for our prison system, because we are required to continue normal operations when we can’t communicate with the other parts of the system. A prison block may not be able to talk to the Registration Office, but that doesn’t mean that it can error. Operations must continue normally.

So RPC over HTTPS is out, but what about messaging? We can use a queue to dispatch commands / events to the other parties in the system, no?

The answer is that we can do that, but let’s consider how that will work. For events, that would work quite nicely, and have roughly the same model that we have now. In fact, with the RavenDB ETL process and the dissemination protocol in place, that is pretty much what we have. Changes to my local state are pushed via ETL to a shared known location and then distributed to the rest of the system.

But what about commands? A command in the system can be things like “we got a warrant for releasing an inmate”. That is something that must proceed successfully. If there is a communication breakdown, generating an error message is not acceptable, in this case, someone will hand deliver the warrant to the block and get this inmate out of the prison.

In other words, we have the notion of commands flowing in the system, but the same command can also come from the Registration Office or from a phone call or from a lawyer physically showing up at the block and shoving a bunch of papers at the sergeant on duty demanding an immediate release.

All of this leads me to create an data architecture that is based on local data interaction with a backbone to share it, while relying on external channels to route around issues between parts of the system. Let’s consider the case of a releasing an inmate. The normal way it works is that the Registration Office prepare a list of inmates that needs to be released today.

Here is one such document, which is created on the Registration Office server.

This then flow via ETL to the rest of the prison. Each block will get the list of inmates that they need to process for release and the Command & Control Center is in charge that they are processed and released properly. I’ll have a separate post to talk about such workflows, because they are interesting, but the key here is that we don’t actually have a command being sent.

Instead, we create a document starting the workflow for release. This is exactly the same as an event being published and it will be received by the other parties in the prison. At this point, each party is going to do their own thing. For example, the Command & Control Center will verify that the bus to the nearest town is ordered and each block’s sergeant is in charge of getting the inmates processed out of the block and into the hand of the Registration Office, which will handle things such as returning personal effects, second verification that the right inmate is released, etc.

Unlike a command, which typically have a success / error code, we use this notion as well as timed alerts to verify that things happen. In other words, the Command & Control Center will be alerted if the inmate isn’t out of the prison by 10:00 AM and will take steps to check why that has happened. The Registration Office will also do its own checks at the close of the day, to ensure that no inmates are still in the prison when they shouldn’t.

Note that this involves multiple parties cooperating with each other, but none of them explicitly rely on request / response style of communication.

This is message passing, and also event publication, but I find that this is a much more passive manner in which you’ll work. Instead of explicitly interacting with the outside world, you are always operating on your own data, and additional work and tasks to do shows up as it is replicated from the other parts of the system.

Another benefit of this approach is that it also ensures that there are multiple and independent verification steps for most processes, which is a good way to avoid making mistakes with people’s freedom.

Properly getting into jail: The topology of sharing

Wed, 07 Mar 2018 10:00:00 GMT

In the previous post, I talked about the flow of data in the system and introduced the notion of sharing data via explicit ETL processes. The ETL process is an explicit contract of each part of the system that shares some of the data to the outside world. By the way, in the context of the prison system, the outside world is a bit misleading, it might be more accurate to say that we share the data with the other parties inside the prison. Nothing here is actually available outside the prison itself (integration with other systems, such as the Department of Justice, other prisons, etc is an even more complex topic that I don’t know if I’ll be covering. For now, assume that the method of integration if a Fax machine).

Here is the simplest topology that we can build using this approach.

The Registration Office has an ETL process that outputs the data it wants to share with a dedicated database, specially for this purpose. From this public database, we setup replication to dedicated database instances on each of the blocks.

In other words, when the application inside the block wants to access some shared data, it isn’t going to reach over the network and hit the public registration database but use a local instance that the public registration database is replicating to.

Why do we have this behavior? Because we don’t trust the network and must always rely on our own local resources. When the network is available, we’ll get continuous updates via replication, and when it isn’t available, we’ll have the latest information that we possibly could and can still act on that (I’ll have a separate post talking about workflow processes that mitigate eventual consistency and concurrency handling in such a system).

This means that the application can always continue running, regardless of the state of the outside world.

Another option, which is somewhat simpler, is to not have a public registration database, but a database that is there to share the data explicitly between all the different parties, this will look like this:

In this case, each party’s setup includes both the internal data that is required to running the block / office / department in question and a shared database that is used to hold all the data that is shared by all the parties in the prison.

Note that the topology of the shared data is a full mesh. In other words, data that is sent to the shared database from the Registration Office using RavenDB ETL will be sent to all the other parties, this is the same as we had before. However, because we now have a shared database, if the Registration Office cannot talk to Block B, that block will still get all the updates from Block A, including the updates that originated from the Registration Office. This is because RavenDB replication uses the gossip model and can bridge such gaps in network without issue.

This might be a simpler model, because the process of each party publishing information for the consumption of the rest of the prison is simplified, we simply define an ETL process to the shared database and the data will be distributed far and wide, made available to anyone that wants it.

This has the advantage that most of the details of managing services can be deferred to RavenDB. You need to make sure that your ETL processes are contractual, that is, they don’t change the shape or meaning of the data, and that is about it. All data access from the application is made to the local database and there is little need to worry about integration between the various parties, error handling of remote calls, etc.

Properly getting into jail: Data flow

Tue, 06 Mar 2018 08:47:00 GMT

In our prison system, we have a lot of independent parts, by design. Each of them is expected to work independently of the rest of the system, but also cooperate with them. That typically require a particular mindset when designing the application.

Let’s lay out the different aspects of integration between the various pieces, shall we?

Each part of the system should be able to run completely independently from the other pieces.
There are offline methods for communications (literally a guy walking down to the block with a piece of paper) that can be a backup communication plane but can also by pass the system entirely (a warrant being served directly to the block’s officers).
There are two distinct options for communication: Commands (release this inmate, ensure inmate is ready to go to court at date) and notifications (inmates count, status, etc)
We trust, but verify, each piece of information that we receive.

The first point seems like a pretty standard requirement, but in combination of the second point, we get into a particularly interesting issue. We may have the same information entered into the system by multiple parties at different times.

For example, a warrant to release an inmate might be served directly to the block’s officer. The release is processed, and then the warrant arrives to the Registration office, which will also enter it. At some later time, this data is merged and we need to ensure that it make sense to the people reading it.

The offline communication plane is also a very important design considerations for a system that reflects the real world. It means that we don’t have to provide too complex an infrastructure for surviving production. In fact, given the fact that a prison is going to hardly have a top notch technical operations team (they might have a top notch operations team, but they refer to something quite different), we don’t want to build something that rely on good communications.

To make sense of such a system, we need to define data ownership and data flow between the various systems. Because this is a really complex system, we’ll take a few example and analyze them properly.

The legal status of an inmate.
The location of an inmate.

What is the meaning of legal status? It means under what warrant it is in the prison (held until trial, 48 hours hold, got a final judgement). At its simplest, it is what date should this person be released. But in practical terms, this can be much more complex and may have conditions on where this inmate can be held, what they can do, etc.

Everything about the legal status of an inmate is the responsibility of the Registration Office. Any movements of inmates into or out of the prison must go through the Registration Office. Actually, this isn’t quite true. Any movement of an inmate from the responsibility of the prison must go through the Registration Office. But the physical location of the inmate is the responsibility of the block into which the inmate was assigned.

A good example of this would be an inmate that has been hospitalized. They are not physically inside the prison, but the prison is still responsible for them. The Registration Office doesn’t usually care for such details (but sometimes they do, for example, if the inmate has a court date that they’ll miss, they need to notify the court) because there isn’t a change in who is in charge of the inmate.

This is complex, but this is also the real world, and we need to manage this complexity. So, let’s define the ownership rules and data flow behavior:

Legal Status is owned by Registration Office and is being disseminated from there to all interested parties.
The location of an inmate and its current physical status are owned by the blocked it is assigned to and disseminated from there to all interested parties.
The assignment of an inmate to a particular block is also interesting. This piece of information is owned by the Registration Office, but it is not their (sole) decision. This may take a bit of explaining.

The block an inmate is assigned to is determined by a bunch of stuff. What is the legal status of the inmate, what is the previous / expected behavior of this inmate, whatever the inmate needs to be isolated from / together with certain people, information / recommendation from the intelligence office, court decisions, the free space available on each block, the inmate medical status and many other details that are not quite important.

The Registration Office will usually make the initial placement of where an inmate is going to go, but this is not their decision, there is a workflow involved that has input from way too many parties. The official decision is at the hands of the prison commander, but recording this decision (and the data ownership of it) is at the hands of the Registration Office.

Okay, enough domain knowledge, let’s talk about the technical details, shall we? I’m sorry that I have to do such an info dump, and I’m trying to contain it to relevant pieces, but if I don’t include the semantics of what we are doing, it will make very little sense or be extremely artificial.

The legal status of inmates in the Registration Office needs to be sent to other parties in the prison. In particular, all the blocks and the Command & Control Center.

We can deal with this by defining the following RavenDB ETL process from the Registration Office:

What this does is simply define the data that we’ll share to the outside world. If I was building this for real, this will probably be a lot bigger, because an inmate is a really complex topic. What is important here is that we define this as an explicit process. In other words, this is part of the service contract that we have with the outside world. The shape of the data and the way we publish it may only be changed if we contact all parties involved. Practically speaking, this usually means that we can only add data, but never remove any fields from the system.

Note that in this case, we simplify the model as we send it out. The warrants for this inmate aren’t going out, and we just pull the latest status and release dates from the most up to date warrant. This is a good example of how we avoid exposing our internal state to the outside world, giving us the flexibility to change things later.

The question now is where does this data goes to? RavenDB ETL will write the data to an external database, and here we have a few options. First, we can define an ETL target for each of the known parties that want this data (each of the blocks and the Command & Control Center, at this time). But while that would work, it isn’t such a great idea. We’ll have to duplicate the ETL definition for each of those.

A better option is to send the (transformed) data to a dedicated database that will be our integration source. Consider the following example:

In this case, we can have this dedicated public database that exposes all the data that the Registration Office shares with the rest of the world. Any part that wants this information can setup external replication from this database to their own. In this manner, when the Intelligence Office decides to make it known that they also needs to access the inmate registration data, we can just add them as a replication destination for this database.

Another option is not to have each individual party in the prison share its own status, but have a single shared database that each of them write to. This can look like this:

In this case, any party that wants to share data will be writing it to the shared database, and anyone who reads it will have access to it through replication from there. This way, we define a data pipeline of all the shared data in the prison that anyone can hook up to.

This post is getting long enough that I’ll separate the discussion of the actual topology of the data and handling the incoming data to separate posts, stay tuned.

Properly getting into jail: it’s not a crime to be in an invalid state

Fri, 02 Mar 2018 10:00:00 GMT

When you need to write a form, one of the first things that pops to mind is what kind of validations are required. A good example would be the new user registration form:

This is such an ingrained instinct that we typically seek to stop such invalid states as soon as possible. If the business rules says that you cannot have a certain state, it is easiest to just ensure that you can never get into such a state.

If you are working in a system that exists purely within the scope of your application, that might actually be useful property to have. This is certainly something that you want in your code, because it means that you can make assumptions on invariants.

If you have a system that reflects the real world, however, you need to take into account several really annoying things:

Your model of the real world is not accurate, and will never be accurate.
The real world changes, sometimes quite rapidly and in a very annoying fashion.
Sometimes the invariant is real, but you gotta violate it anyway.
If the system doesn’t let the users do their job, they will do the job in spite of your system. That will lead to a system of workarounds. Eventually, you’ll have to support these workarounds. This may result in hair loss and significant amount of aggravation.

Consider the case that the prison in question is a minimum security prison, expected to have white collar inmates that have already been tried. Now you get an inmate that is accused of murder and is currently on trial. (Side note, prisons have a very different structure for people who still undergoing trial, because they aren’t convicted yet and because the amount of pressure that they are under is very high. They’ll typically be held in different facilities and treated differently from inmates that have already been convicted and tried).

What do you think will happen if the system refuses to accept such an inmate into the prison? Well, the guy is right there, and the prison commander has already authorized letting him in. So telling him that he should go back home because they “computer won’t let you in” is not going to be a workable solution.

Instead, we use a very different model for validation and acceptance of data. Instead of stopping the bad input as soon as possible, we raise a flag about it during processing, but we do allow the user to proceed, assuming they explicitly authorize this. It will look like this:

At the same time, we flag such records and require additional review of the data.

In most cases, by the way, the fact that the inmate is in residence is not something that can be ignored and will be in all reports on the state of the prison until the state changes (transferred, verdict given, etc).

This kind of thinking is important, because it means that the software intrinsically does not trust the data, but continue runs validation on it. This is a good practice to be in when dealing with systems that reflect the real world.

This does lead to an interesting question, where do we run these validations, and when?

The ground rules for a good application is that it is like an amoeba, it may have multiple locations that it accepts input, but that is the only way to push data in, through well defined channels.

This is another way of saying that we don’t allow someone else to go and poke around in our database behind our back.

Any time that we accept new data, regardless of the channel it arrives in, we run it through the validation pipeline. And this validation pipeline can add (or mark as obsolete) validation issues that should be brought to the attention of the people in charge.

Note that these sort of validations are almost always very much business rules, not type issues. If someone’s birthday is in the future, you can feel free to very easily reject that data as purely invalid. But if someone’s release date is in the past, however, you might still need to accept them as an inmate (the paperwork are really in the mail, sometimes).

I think I’ll need another post just to talk about how to implement these validation rules and behaviors, but that is still down the line. The next post topic is going to be data flow between the different systems that we talked about so far.

Properly getting into jail: Physical architecture

Thu, 01 Mar 2018 10:00:00 GMT

When building a system for a prison you need to consider a few more levels of high availably than usual. In particular, you might need to operate while there are riots in progress, which may be prison wide, in a single block or section or in general while in a partial failure mode. Regardless of what is going on, you want maximum survivability for the system. If there is a riot in the next block, I still want to be able to do a proper count of the inmates in this block, for example. So each block must be able to operate independently from the others. For that matter, the command & control center must always be operational and so does the medical ward, the registration (who is in the prison right now is something that is important to know especially when there are issues.

All of this means that we can’t rely on any centralized architecture and that we want maximum separation between the different pieces. The reason this post is titled physical architecture is just that, we are talking about physical machines that reside some distance from one another. Now, granted, prisons are, by their very nature, limited in size, so we don’t have to worry too much about the latency of calls. However, we are worried about other aspects. We have to assume low to no maintenance and the chance of riots / floods and possible locusts.

The best case scenario for us is that each server will be installed on its own block, with a UPS if we are lucky. The whole thing residing in a locked cupboard in the block’s sergeant's office. Given typical conditions, the cupboard is also were the teapot and coffeemaker will reside, and we’ll have to worry about our server going doing because someone spilled a whole pot of tea on the machine. But I’ll leave completely fictional heart attack inducing events to another day and focus on the software architecture.

We need to be able to operate each block independently of other blocks and from the central prison administration. That means that we will use an architecture that assumes that we are an island. All the applications that are used will only talk to a local database, and cannot assume any other connectivity being available.

Instead, we’ll build explicit data flow channels between the different parts of the system and handle that part explicitly.

It is important to note that this type of architecture also has another major benefit, it cushions certain type of errors. If an inmate managed to trip on a wire and cut the network cable leading to the block, we want to still be able to operate normally for all / most usual things. This means that we can summon the network guy to come and fix things in the morning, rather than have them come up now. That is the kind of architectural results that you like if you are one of the guys who might be called in such a case.

The block server will hold:

The inmates that are in the block and well as their record in the block (disciplinary actions, privileges withheld / given, notes, etc).
Records of counting of inmates.
Log of actions happening in the block (Sargent's diary).
Tasks and operations required (sending inmates to court, scheduled work parties, cell searches, etc)

That kind of data is essential for the routine operation of the block and should always be available.

At the prison’s level, you have all kind of departments and the kind of data they need to keep:

Medical – medical history and other medical stuff. This is a separate system, because there is nothing unique in a prison medical ward that require anything special other than the bars on the windows and better locks on the drug cabinets.
Registration – all the inmates in the prison as well as all their holding documents. These are the documents and authorization to withhold someone’s freedom, and it is stored and managed by this department. This is also the department that can generate work orders (send this inmate to this court) for the rest of the prison and is the primary outside contact.
Command & control center – has a view of everything and is in charge of tracking operations, but the only action it takes is to pass that information onward.

I’ll speak more about each of these departments at length in the future, but for now I want to focus on something that is very interesting. None of these departments actually need to talk to anyone else to do most of its job.

The C&C center needs to get reports from the blocks about counts, that inmates were sent to court, that the inmates that were supposed to be freed were actually freed, etc. But none of that has to happen through regular channels. In fact, in many cases, they’re multiple channels that are used on an ongoing basis.

In addition to having the block’s server notify the C&C systems that the counts were completed, the sergeant will also notify C&C directly via a phone call about the count and how many inmates are currently in residence in the block.

The prison software needs to record all of that. In fact, even though it looks like it is doing the same work twice, we need it to happen, for the same reason we have double entry book keeping. It helps catch issues and omissions.

So that leads to another architectural tenant for the system. Not only are the system made of physically separated components, but we are also considering each portion of the system as a consistency boundary.

I hesitated here with regards to the naming. I almost went with a security boundary, but that isn’t quite right. It isn’t that the other side is not trusted in the classic sense. They are trusted, but they aren’t verified. We’ll need to double check all such data as it goes into the system. But that leads to the next big challenge, how are we going to handle the flow of data and verification of the data across the system as a whole.

I think this post is long enough as it is, so I’ll keep that for the next one.

Properly getting into jail: Counting Inmates and other hard problems

Wed, 28 Feb 2018 10:00:00 GMT

In prison, the notion of counting people is sacred. This is probably because the whole point of having a prison is to keep the people you put in there inside, and that means that you count inmates, multiple times a day.

The dirty secret, however, is that you almost never get to that perfect occupancy number, where all the inmates that are registered to a particular block are actually in that block. In most cases, you have at least a few that are outside the block (court dates, medical issues with offsite care, visitations, outside work, etc).

So the key here is not just to count the inmates, but to also account for them. Let’s consider how this will look like in the user interface for counting a couple of cells, shall we?

This works, but it isn’t a good idea. Named counts are usually reserved for the first / last counts of the day. During the day, because it is so frequent to have inmates in and out of their assigned location, you’ll usually do things differently. You’ll have a total count of inmates in the block, and a list of the exceptions. That would look something like this:

You have the number of inmates in the block, how many are expected to be there, the actual count as verified by the sergeant’s signature and the named list of inmates that are not currently in the block. You might have noticed that we are carefully tracking who is responsible for any inmate that it currently out of the block. This is because that matters (for a whole host of legal and cover your ass reasons).

So how would we build such a system?

To start with, we need to talk about the current component or service that we are building. The notion of counting is typically done at the block level, so we’ll start by modeling things there. We’ll have the Block Service, which is in charge of managing anything that is going on inside the block.

A block is composed of:

Cells, to which inmates are assigned. This is typically an internal division only that has no real business meaning.
Inmates, which are quite important.
Staff, which is probably a separate issue entirely, but is quite important for thing such as having enough people at hand to do things like actually run the block.

In terms of the actual operations we need to do, the block is managed by at least a single Sargent per shift and multiple guards. The Sargent is responsible for handling incoming inmates, counting all inmates multiple times a day and other things that we won’t be tracking on a computer system. The guards will mostly interact with the system when they need to check an inmate out for whatever reason (such as taking them to a checkup by a nurse).

With all of this information, we can now model the data we have for a block. Here is the most important document we have, the block’s population. There are a few things here that are worth exploring in the design of the document:

First, we have a separate document per date, recording the state of the block’s population at that time. This is important, because we need to be able to go back in time and track such things. You can also see that the document contains a lot of data that has both the name and id. Why is that?

The information recorded on this document is the data as it was at the time of this document’s creation. Later changes do no apply, by design, since we need to see keep it at the state it was at the time. It might be easier to look at things in code:

The most important thing here is the notion of the Log, which records every incoming and outgoing inmate from the block.

In addition to the daily’s block population document, we also have three to five counting documents, which are built on top of it. These reflect the actual counts made plus the listing of inmates that aren’t currently in the block and why.

And that is quite enough about the prison’s life. This give us sufficient level of details that we can now work with. The next post will talk about how the physical architecture and data flow in the prison.

Properly getting into jail: Introduction & architecture

Tue, 27 Feb 2018 10:00:00 GMT

I have been writing about the internals of RavenDB for quite a while, and it is fascinating in many respects, but it gets tiring to keep thinking about bits & bytes all the time. I’m missing writing some of the more high level stuff, in particular about software architecture. So I thought that I would take the time to resurrect a very old post series of mine, Macto. I actually have quite a few posts about this, but they are all close to a decade old, so I might as well start from scratch.

Macto is a sample app for managing a prison. This is one of the few areas in which I can be an expert on the business requirements that isn’t utterly boring (I don’t want to do yet another E-Commerce stuff). To keep up with the times, Macto is going to be written with a Micro service architecture, and just for fun, I might be doing that in multiple languages and platforms, because prison is not fun, and neither should be working on it’s computing systems . Oh, and there is also the real world thing, I guess.

Since 2009, I have pretty much given up on building anything UI wise, so I’m going to show a few mockup screens, but the idea is that I’m going to be looking only at backend code, with another team actually doing the user interface.

The first thing to start with, I guess, is to paint the overall architecture of the system. Prisons are pretty rigid systems, as you might expect, but there are a lot of interconnected parts. In order to properly build a system to manage a prison we need to be able to answer what is going on from multiple points of view and from very different perspectives.

The prison commander cares about such things as the inmate’s count (the job is to keep them all hale and accounted for). The individual guard care about the particular set of cells they are assigned to and the inmates in them. The guys at Registrations care about the legal details of having a lawful warrant for holding an inmate in prison as well as when an inmate should be released. Intelligence cares about the connections between inmates, the kind of heads up that came through channels and what kind of actions should be taken as a result. Medical needs to verify that incoming inmates are fit to be held in the facility and Transfers needs to ensure that any movements of inmates outside the prison will complete successfully (as in, you got them out, you also gotta bring them all in).

Each of those pieces interact with others in interesting and complex ways. For example, incoming inmates needs to go through Registration for legal paperwork, Medical for certification, Intelligence for verification and then assigned to the proper block. Once they are accepted into the prison, they are the charge of the particular block they are assigned to and rarely need to interact with the rest of the prison unless something extraordinary happens (visits, court, sickness, etc).

When thinking about the software architecture of such a system, the most important rule to remember is that we want the system to be used, this means that we really need to plan for what people are actually doing (in abstract of what they should be doing) and to help them do things, rather than hinder them. In most places, all these details are done with pen & paper, and it works, so our system will have to offer something more. Not to the prison administration, but to the actual people going about their work with the inmates.

From a software architecture point of view, we are going to model the system as a set of independent services that will each have the role of one of the departments inside the prison. The current term in micro-services, but in real systems, they are not so micro, so we might need to repeatedly break things apart until we get to a level in which things make sense in isolation.

A lot of the complexity is involved in managing such a system is in the flow of information across the system. In a prison, this is the responsibility of the Command & Control Center (C3, from now on) which is in charge of coordination and monitoring of actions across the board. This also work very closely with the heads of all departments and the prison commander as well as most other external parties, but it generally does nothing on its own.

I think that this is enough of an intro, and we’ll get right on to things in the next post, where we’ll talk about Counting Inmates.

Macto: Looking at warrants

Wed, 17 Aug 2011 09:00:00 GMT

After spending so much time talking about how important Warrants are, it is actually surprising to see the UI for a warrant:

It is pretty simple, because from a data entry perspective, there isn’t really much to it. It is the effects of the Warrants that make it such an interesting concept. One thing to note here is that the date that we care about for the Warrant isn’t the date it was issued, but when the guy was actually arrested, that is where the clock starts ticking.

Adding a Warrant is also not that complex from a data entry perspective:

As you can see, thus far, it is pretty simple. But when you click the Finish button the complexity starts.

We need to check that the Warrant is valid (issued by someone with the authority to do so), and then calculate the new duration for the Inmate.

And that is enough for today, I just wrote ~10 posts on Macto in the last 24 hours, it is time to do something else.

Macto: Talking to nasty people

Mon, 15 Aug 2011 09:00:00 GMT

Well, so far we have looked at the main screen and the Counting screen, it is time we introduce ourselves to the Inmate screen as well:

As you can see, there isn’t a lot going on here. We have the Inmate’s Id, name and location (including noting where he is located, if he is known to be outside the prison). We don’t usually care for things like detailed information in the normal course of things (the Inamte’s national id, for example). That information is important, but usually not relevant, we relegate it to a separate screen.

The dates for incarceration and scheduled release are also important, but they aren’t available for editing, they are there only for information purposes.

The note is there to make sure that highly important information (such as whatever the Inmate is suicidal / flight risk) would be clearly available. The same is true for the Cliff Notes version of the Record.

It is there not for casual use, but to ensure that pertinent information is available. Most importantly, note the last line there. Which means that if this Inmate is about to be released, we have to notify someone and get their approval for that. Well, approval is a strong word, we notify them that they need to give us a new Warrant for the Inmate, but we can delay releasing him (say, until midnight the day he is to be released) waiting for that Warrant.

Warrants are important, and you can see the ones that this guy have listed here. The last Warrant is the important one, the others are shown for completion sake and to ensure that we have continuous Warrants.

There are also several actions that we can do to an Inmate. We can Transfer him to another prison, Release him or Add a Warrant. Each of those is a complex process of its own, and I’ll discuss them later on.

Macto: Counting is The Holy Grail

Thu, 11 Aug 2011 09:00:00 GMT

I might have mentioned before that Counting is somewhat important in prison. That is about as accurate as saying that you would somewhat prefer it to keep breathing. Counting is the heartbeat of the prison, the thing that the entire operation revolves around.

Here we can see the counting screen:

You can see that we have an Open Count here, that is a count that is still in progress. Some cell blocks have reported their counts, and some are still in the process of making the count.

A Count is Closed when we know where are the Inmates are. (When the two numbers line up properly). You probably noticed that there are two ways to report a count, the first is for Inmates who are outside the prison (court, hospital, etc). Because they are outside the prison, we track them by name. For internal counts, we don’t really care about names, just the numbers.

There is another type of a Count, a Named Count, which is a process that happen very rarely, but is usually used to reconcile what is in the computer and what is actually in the cells.

It is important to understand the “Officer in charge” field, basically, it is the guy who has the legal responsibility and is signing off on those numbers. In other words, if there is something wrong, that guy is going to take a hard fall.

Macto: Getting Started, you never forget your first Inmate

Wed, 10 Aug 2011 09:00:00 GMT

People always love to start with CRUD, but it is almost never that simple. In this post, we will review the process required to accept an Inmate into the prison.

That process is composed of the following parts:

Identification
- Who is the guy?
- Id number
- Names
- Photo
- If we can’t identify / refuse to identify, we still need to be able to accept him.
Lawful chain of incarceration
- Go over all the documents for his arrest
- Ensure that they are all in order
- Ensure that they are continuous and valid
- Check if there are any urgent things to do with him. For example, he may need to be at court today or the next day.
Medical exam
- Is it okay to hold the guy in prison?
- If he is not healthy, can we take care of him in prison?
- Does he require hospitalization?
- Does he require medicine / treatment?
- Are there any medical consideration into where to put him?
Intelligence
- Interviewing the guy
- Report interesting details that are known about him
Acceptability
- Does he fit the level of Inmates we can accept? We usually don’t put murderers in minimum security prisons, for example.
- Does he have any medical reason to reject him?
- Are there any problems with the incarceration documents?
- Is there any intelligence warning about the guy?
Placement
- Decide where to put the Inmate
- What type of an Inmate is he? (Just arrested, sentenced, sentenced for a long period, etc)
- Why is he in prison for?
- What kind is he? (You want to avoid Inmate infighting, it creates paperwork, so you avoid putting them in conflict if possible)
- Where there is room available?

Another important aspect to remember is that while we are allowed to reject invalid input (for example, we are allowed to say that the id number has to consist of only numeric characters), we are not allowed to reject input that is wrong.

What do I mean by that. Let us say that we have an Inmate at the door, and he doesn’t have his incarceration paperwork in order (well, not he, whoever brought him in, but you get the point). That means that legally, we can’t hold him. But Macto isn’t where things are actually happening, it is merely a support system that tracks what is going on in the real world. And the prison commander can decide to accept that guy anyway (say, because the paperwork in en route), and we have to allow for that. If we try to stop people from doing this, it is going to be worked around, and we don’t want that. The system is allowed, even encouraged, to warn the users when they are doing something wrong, but it cannot block it.

The first part, Identification, is actually pretty easy, all told. This is fairly simple data entry process. We’ll want to do some checkups on the data, such as that the id number is valid or to check the id against the name, etc. But we basically have to have some level of trust in the documents that we have. You usually don’t have an arrest warrant for “tall guy in brown shirt”. If we find any problems there, we can Flag the Dossier as a potentially fraudulent name. This is also the stage where we want to check if the Inmate is a returned visit, and bring the life the old Dossier.

The second part is more complex, because there are many different types of Warrants, each with their own implications. Arrest Warrant is valid for 24 hours, Remand Warrant is good until sentencing, etc. We need to input all of those Warrants, ensure that they are consistent, valid and continuous. If there is a problem with that, we need to Flag the Dossier, but we can’t reject it. We will discuss this in more detail in the next post.

The third part is basically external to Macto, we may need to provide the Inmate ID, for correlation purposes, but nothing beyond that. We do need to get approval from the doctor that the Inmate is in an OK condition to be held in the prison. That does get recorded in Macto.

The forth part is again, external to us. Usually any information is classified and wouldn’t appear in Macto. We may get some intelligence brief about the guy, but usually we won’t.

The fifth part is important, this is where we actually take legal and physical ownership for the Inmate. Up until that point, we had him in our hands, but we weren’t responsible for him. Accepting the Inmate is a simple matter if everything is good, but if the Dossier was Flagged, we might need approval from the officer in charge. Accepting an Inmate means that he is added to the prison’s Roster.

The sixth part is pretty much “where do I have a spare bed”, after which he is added to the Roster of the cell block he is now in care of.

It is important to note that Placement always happens. Even if immediately after Accepting an Inmate you rushed him to the hospital, that Inmate still has to be assigned to a cell block, because that assignment means that the cell block commander is in charge of him. That way we avoid potential mishaps when an Inmate is assigned to no one, doesn’t get Counted.

Okay, I think that this is enough for now, in the next post, we will discuss what exactly goes on in the second part, it is a pretty complex piece, and it deserve its own post.

Macto: The Main Screen

Mon, 08 Aug 2011 09:00:00 GMT

I usually like to think about the responsibilities of the system by showing up the UI. It is a great way to communicate with both customers and developers.

Here is the main screen for the application:

This is actually a bad place to start with, in terms of coding start points, because it requires so many other things as well. This particular screen is likely to be viewed pretty much everywhere, this is what the prison commanders and the cell blocks commanders have at their desktop, what the officers are using to do their daily work (usually for Counting, admittedly).

Even before we can start , it reveals quite a lot about the actual way things work. In this screen, we can see that we have Flagged Dossiers, those are Inmates that have some problem with their Dossier. We can accept Inmates with problematic Dossiers, but we want to fix that as soon as possible, so we make this something that is very much front and center.

The “Action required” section detail Inmates that we have to take some action about. Whatever it is a court date that this inmate have to be at or his sentence is ending or a warrant that need extending.

Finally, and most importantly, we have the counts, which are the most important thing in the prison. You can see that the numbers at the bottom line up. If they don’t, we have A Problem.

Macto: Warrants are for fools

Wed, 03 Aug 2011 09:00:00 GMT

Warrants are kinda important in a prison. They are the legal authority to limit someone’s freedom. In wouldn’t be overstating the fact in saying that Warrants are one of the major factors being managed in Macto.

There are all kind of Warrants in existence. To list just a few of them:

Arrest Warrant – Issued by an officer, generally hold for 24 hours only.
Detention Warrant – Issued by the court, generally for a short amount of time, up to a few weeks, in most cases.
Remand Warrant – Issued by the court, generally instructing the prison to hold the Inmate in custody until sentencing (not limited in time).
Sentencing Warrant – Issued by the court, specifying the total time that an Inmate is to be incarcerated.

There are other warrants, such as an Court Arrest Warrant, for example, but for the purpose of Macto, we won’t get into those. The type of activity currently required by the prison doesn’t really need them, but that might change in the future.

There is also another type of Warrant available, it is Whatever The Judge Said Warrant, or as the lawyers call is Mandamus Warrant. It is basically an instruction to do something, and it can be just about anything. From letting the Inmate to call his wife to putting him in a different cell or transferring him to a different prison to command special food / treatment to… Well, there is a reason I call it Whatever The Judge Said.

The rules for Warrants for incarceration are pretty simple. Each warrant type has an issuer (Arrest Warrants can only be given by Officers of rank Captain and above) for a certain duration (which in some cases, may be limited, such as the 24 hour limit for Arrest Warrants). Depending on the type of Warrant, it can be in Hours, Days, Months or Years. The units in which a warrant is specified are very important. In particular, there is a difference between 30 days incarceration and 1 month incarceration, for example. And hourly Warrants requires that by the time the Warrant expire, you are either got a new one at court or let the Inmate go.

The last issued Warrant is always the one that is valid, and all Warrants must be continuous. Gaps in the middle are considered to be a Very Bad thing.

Macto: Non functional concerns, you are a legal system

Mon, 01 Aug 2011 09:00:00 GMT

Macto is a system that operates in a highly legislative environment. As such, we have to be prepared for the court to ask us to show our records about a particular Inmate. Part of that is ensuring that we preserve the history of the Inmate’s Dossier. An example where this would be relevant is when a lawyer contend the legality of incarcerating the Inmate. You have to show not only that you have legal authority to incarcerate the guy, you also have to show that you had that authority continuously throughout the incarceration period.

A typical case where there is a problem is shown below:

27 June 2011 20:52 – Arrest by Sargent Azulay for car vandalizing.
29 June 2011 09:15 – Detention, 8 days by Judge Judy
5 July 2011 – Remanded in Custody by Judge Thachil Oti
14 Aug 2011 – Sentenced, 3 months by Judge Koev Li
27 Sep 2011 – Released at end of sentence

Do you see the problem? You probably don’t, but for me, it shouts. The issue is that an Arrest is only valid for 24 hours. Because of the gap in the incarceration warrants, a lawyer can usually get an Inmate out.

That means that part of what the system has to do is to be able to say not only what the current state, but what was the state at any given point in time. Those are usually called Temporal Systems, or Append Only systems, since you are not allowed to make modifications existing data, only create new data.

They also tend to be quite hard to work with, but this is still isn’t a post about the technical stuff, so we will let it go until we get to the good parts.