Ayende @ Rahien

My name is Oren Eini
Founder of Hibernating Rhinos LTD and RavenDB.
You can reach me by email or phone:


+972 52-548-6969

, @ Q j

Posts: 6,591 | Comments: 48,240

filter by tags archive

Properly getting into jailThe workflow of getting an inmate released

time to read 6 min | 1025 words

imageThe whole point of having a prison is ensuring that the inmates are staying in. As such, the process of actually getting an inmate out of the prison is quite involved. I’m using the release workflow as an obvious example, but there are many such workflows inside the prison that are similar. The inmate’s intake workflow, or disciplinary workflows or even just inmate’s transfers which can be quite complex (and typically also happen in big batches).

In the previous post in this series, I talked briefly about the idea of workflows as an event publication with multiple signatories to the workflow operation. If this make no sense, let me try to explain.

The trigger for starting an inmate’s release can be the acceptance of a warrant for immediate release, the expiration of the warrant to hold the inmate. There are several other variants (transfer to another facility, death of inmate, escape of inmate, etc) that are related but are not too relevant for the discussion, so I’ll just talk about the immediate release and the warrant expiration.

For warrant expiration, the workflow starts from the Registration Office publishing the list of inmates that are supposed to be released today. In technical terms, they create a series of workflow documents that are in the Open state and this gets disseminated to the rest of the system. Immediate release warrant is usually served to the Registration Office, but may also be served directly to the block’s sergeant.

If the Registration Office got the immediate release warrant, things are identical as the usual scheduled release (except that this might be served at any time). However, if this is served to the block’s sergeant, things are more interesting. At this point, it is the block that will initiate the workflow, but the overall responsibility for verifying the warrant and ensuring the actual release is still on the Registration Office.

As an aside, the notion of "who owns this” is quite important in the prison. Mostly because if you mess up, there might be consequences. This can be holding an inmate past the due date (bad, can result in damages paid to the inmate and career consequences to the person who messed up) to releasing an inmate that wasn’t supposed to be released (really bad, sometimes it is not possible / feasible to hold them again, requires court approval, may end up in jail time for the person who messed up and also result in possible dangerous inmate being released). So the idea is that at any time, there is an owner for this process and a clear finger pointing at that person, “You’re to blame”.

Because of this, there are typically multiple steps in the release process. Consider the simple scenario of a scheduled release, we have:

  • Registering the inmate on the “to be released” list.
  • Notifying external and internal parties about this inmate’s pending release. For example, an inmate might have to go through a parole officer before actual release. This is done by sending the “to be released” list several days in advanced and getting at least an implicit agreement (by not vetoing / modifying the process) by external parties.
  • Verification that there are no explicit holds on the inmate. In the case of an inmate that is supposed to be deported, the inmate’s file will have an explicit “deport on release” which typically require coordinating with the border police to handle that. So the inmate can’t just be shoved out the door but handed off to someone.
  • Identification of the inmate at the block level. This is typically done on the sergeant’s level and then by an officer (preferably from the same block) that are familiar with the inmate and can validate that this is indeed the one to be released.
  • Checking out the inmate from the block level to the prison’s level. This explicitly remove the inmate from the block’s responsibility once the inmate has been handled off to the release queue.
  • Identification of the inmate by the Registration Office’s officer. This is a second verification that is done to ensure that there hasn’t been a mix up again.
  • Verification of the warrant to release the inmate and that there are no newer warrants that are in effect.
  • Returning of personal affects and getting the inmate’s signature that everything was properly returned.
  • Checking the inmate out of the prison, this step explicitly ends the period in which the now ex-inmate is held in prison.
  • Actually getting the newly released ex-inmate out of the prison. This can be to a family member at the gate or to a bus to the nearest city, etc.

I’m probably forgetting a few details in the middle, and there are branches for each and every one of these steps.

In terms of the technical details of how this works. The workflow document is being distributed throughout the system, and then various parties are now in charge of actually completing the various tasks in the workflow. For example, this may mean checking out the inmate’s personal affects from the safe in the morning, preparing for the release. So the steps aren’t necessarily in sequence or ordered.

The important thing is that the workflow document is published, and is now tracked. Typically, the release process is tracked by the Registration Office and the Command & Control Center. The Command & Control Center will typically be involved early in the process if an inmate isn’t released by the typical time, and the close of day process for the Registration Office includes verification that there aren’t any still pending inmate releases.

At any given point, we can track the process of the workflow from any point in the system (remember, changes are made locally and then distributed to the rest of the system). In there is a communication issue between different parts of the prison, that will typically show up as an alert that a particular workflow hasn’t been properly completed in the allotted time. At this point, external channel (walking & talking, usually) is used to verify the status of this particular inmate release.

Properly getting into jailServices with data sharing instead of RPC or messaging

time to read 5 min | 857 words

imageThe design of Macto (that is this prison management application I’m discussing) so far seems pretty strange. Some of the constraints are common, the desire for resiliency and being able to have each independent portion of the system work in isolation. However, the way we go about handling this is strange.

I looked up the definition of a micro services architecture and I got to Martin Fowler, saying:

..the microservice architectural style [1] is an approach to developing a single application as a suite of small services, each running in its own process and communicating with lightweight mechanisms, often an HTTP resource API.

This wouldn’t work for our prison system, because we are required to continue normal operations when we can’t communicate with the other parts of the system. A prison block may not be able to talk to the Registration Office, but that doesn’t mean that it can error. Operations must continue normally.

So RPC over HTTPS is out, but what about messaging? We can use a queue to dispatch commands / events to the other parties in the system, no?

The answer is that we can do that, but let’s consider how that will work. For events, that would work quite nicely, and have roughly the same model that we have now. In fact, with the RavenDB ETL process and the dissemination protocol in place, that is pretty much what we have. Changes to my local state are pushed via ETL to a shared known location and then distributed to the rest of the system.

But what about commands? A command in the system can be things like “we got a warrant for releasing an inmate”. That is something that must proceed successfully. If there is a communication breakdown, generating an error message is not acceptable, in this case, someone will hand deliver the warrant to the block and get this inmate out of the prison.

In other words, we have the notion of commands flowing in the system, but the same command can also come from the Registration Office or from a phone call or from a lawyer physically showing up at the block and shoving a bunch of papers at the sergeant on duty demanding an immediate release.

All of this leads me to create an data architecture that is based on local data interaction with a backbone to share it, while relying on external channels to route around issues between parts of the system. Let’s consider the case of a releasing an inmate. The normal way it works is that the Registration Office prepare a list of inmates that needs to be released today.

Here is one such document, which is created on the Registration Office server.


This then flow via ETL to the rest of the prison. Each block will get the list of inmates that they need to process for release and the Command & Control Center is in charge that they are processed and released properly. I’ll have a separate post to talk about such workflows, because they are interesting, but the key here is that we don’t actually have a command being sent.

Instead, we create a document starting the workflow for release. This is exactly the same as an event being published and it will be received by the other parties in the prison. At this point, each party is going to do their own thing. For example, the Command & Control Center will verify that the bus to the nearest town is ordered and each block’s sergeant is in charge of getting the inmates processed out of the block and into the hand of the Registration Office, which will handle things such as returning personal effects, second verification that the right inmate is released, etc.

Unlike a command, which typically have a success / error code, we use this notion as well as timed alerts to verify that things happen. In other words, the Command & Control Center will be alerted if the inmate isn’t out of the prison by 10:00 AM and will take steps to check why that has happened. The Registration Office will also do its own checks at the close of the day, to ensure that no inmates are still in the prison when they shouldn’t.

Note that this involves multiple parties cooperating with each other, but none of them explicitly rely on request / response style of communication.

This is message passing, and also event publication, but I find that this is a much more passive manner in which you’ll work. Instead of explicitly interacting with the outside world, you are always operating on your own data, and additional work and tasks to do shows up as it is replicated from the other parts of the system.

Another benefit of this approach is that it also ensures that there are multiple and independent verification steps for most processes, which is a good way to avoid making mistakes with people’s freedom.

Properly getting into jailThe topology of sharing

time to read 4 min | 669 words


In the previous post, I talked about the flow of data in the system and introduced the notion of sharing data via explicit ETL processes. The ETL process is an explicit contract of each part of the system that shares some of the data to the outside world. By the way, in the context of the prison system, the outside world is a bit misleading, it might be more accurate to say that we share the data with the other parties inside the prison. Nothing here is actually available outside the prison itself (integration with other systems, such as the Department of Justice, other prisons, etc is an even more complex topic that I don’t know if I’ll be covering. For now, assume that the method of integration if a Fax machine).

Here is the simplest topology that we can build using this approach.


The Registration Office has an ETL process that outputs the data it wants to share with a dedicated database, specially for this purpose. From this public database, we setup replication to dedicated database instances on each of the blocks.

In other words, when the application inside the block wants to access some shared data, it isn’t going to reach over the network and hit the public registration database but use a local instance that the public registration database is replicating to.

Why do we have this behavior? Because we don’t trust the network and must always rely on our own local resources. When the network is available, we’ll get continuous updates via replication, and when it isn’t available, we’ll have the latest information that we possibly could and can still act on that (I’ll have a separate post talking about workflow processes that mitigate eventual consistency and concurrency handling in such a system).

This means that the application can always continue running, regardless of the state of the outside world.

Another option, which is somewhat simpler, is to not have a public registration database, but a database that is there to share the data explicitly between all the different parties, this will look like this:


In this case, each party’s setup includes both the internal data that is required to running the block / office / department in question and a shared database that is used to hold all the data that is shared by all the parties in the prison.

Note that the topology of the shared data is a full mesh. In other words, data that is sent to the shared database from the Registration Office using RavenDB ETL will be sent to all the other parties, this is the same as we had before. However, because we now have a shared database, if the Registration Office cannot talk to Block B, that block will still get all the updates from Block A, including the updates that originated from the Registration Office. This is because RavenDB replication uses the gossip model and can bridge such gaps in network without issue.

This might be a simpler model, because the process of each party publishing information for the consumption of the rest of the prison is simplified, we simply define an ETL process to the shared database and the data will be distributed far and wide, made available to anyone that wants it.

This has the advantage that most of the details of managing services can be deferred to RavenDB. You need to make sure that your ETL processes are contractual, that is, they don’t change the shape or meaning of the data, and that is about it. All data access from the application is made to the local database and there is little need to worry about integration between the various parties, error handling of remote calls, etc.

Properly getting into jailData flow

time to read 8 min | 1444 words

imageIn our prison system, we have a lot of independent parts, by design. Each of them is expected to work independently of the rest of the system, but also cooperate with them. That typically require a particular mindset when designing the application.

Let’s lay out the different aspects of integration between the various pieces, shall we?

  1. Each part of the system should be able to run completely independently from the other pieces.
  2. There are offline methods for communications (literally a guy walking down to the block with a piece of paper) that can be a backup communication plane but can also by pass the system entirely (a warrant being served directly to the block’s officers).
  3. There are two distinct options for communication: Commands (release this inmate, ensure inmate is ready to go to court at date) and notifications (inmates count, status, etc)
  4. We trust, but verify, each piece of information that we receive.

The first point seems like a pretty standard requirement, but in combination of the second point, we get into a particularly interesting issue. We may have the same information entered into the system by multiple parties at different times.

For example, a warrant to release an inmate might be served directly to the block’s officer. The release is processed, and then the warrant arrives to the Registration office, which will also enter it. At some later time, this data is merged and we need to ensure that it make sense to the people reading it.

The offline communication plane is also a very important design considerations for a system that reflects the real world. It means that we don’t have to provide too complex an infrastructure for surviving production. In fact, given the fact that a prison is going to hardly have a top notch technical operations team (they might have a top notch operations team, but they refer to something quite different), we don’t want to build something that rely on good communications.

To make sense of such a system, we need to define data ownership and data flow between the various systems. Because this is a really complex system, we’ll take a few example and analyze them properly.

  • The legal status of an inmate.
  • The location of an inmate.

What is the meaning of legal status? It means under what warrant it is in the prison (held until trial, 48 hours hold, got a final judgement). At its simplest, it is what date should this person be released. But in practical terms, this can be much more complex and may have conditions on where this inmate can be held, what they can do, etc.

Everything about the legal status of an inmate is the responsibility of the Registration Office. Any movements of inmates into or out of the prison must go through the Registration Office. Actually, this isn’t quite true. Any movement of an inmate from the responsibility of the prison must go through the Registration Office. But the physical location of the inmate is the responsibility of the block into which the inmate was assigned.

A good example of this would be an inmate that has been hospitalized. They are not physically inside the prison, but the prison is still responsible for them. The Registration Office doesn’t usually care for such details (but sometimes they do, for example, if the inmate has a court date that they’ll miss, they need to notify the court) because there isn’t a change in who is in charge of the inmate.

This is complex, but this is also the real world, and we need to manage this complexity. So, let’s define the ownership rules and data flow behavior:

  • Legal Status is owned by Registration Office and is being disseminated from there to all interested parties.
  • The location of an inmate and its current physical status are owned by the blocked it is assigned to and disseminated from there to all interested parties.
  • The assignment of an inmate to a particular block is also interesting. This piece of information is owned by the Registration Office, but it is not their (sole) decision. This may take a bit of explaining.

The block an inmate is assigned to is determined by a bunch of stuff. What is the legal status of the inmate, what is the previous / expected behavior of this inmate, whatever the inmate needs to be isolated from / together with certain people, information / recommendation from the intelligence office, court decisions, the free space available on each block, the inmate medical status and many other details that are not quite important.

The Registration Office will usually make the initial placement of where an inmate is going to go, but this is not their decision, there is a workflow involved that has input from way too many parties. The official decision is at the hands of the prison commander, but recording this decision (and the data ownership of it) is at the hands of the Registration Office.

Okay, enough domain knowledge, let’s talk about the technical details, shall we? I’m sorry that I have to do such an info dump, and I’m trying to contain it to relevant pieces, but if I don’t include the semantics of what we are doing, it will make very little sense or be extremely artificial.

The legal status of inmates in the Registration Office needs to be sent to other parties in the prison. In particular, all the blocks and the Command & Control Center.

We can deal with this by defining the following RavenDB ETL process from the Registration Office:


What this does is simply define the data that we’ll share to the outside world. If I was building this for real, this will probably be a lot bigger, because an inmate is a really complex topic. What is important here is that we define this as an explicit process. In other words, this is part of the service contract that we have with the outside world. The shape of the data and the way we publish it may only be changed if we contact all parties involved. Practically speaking, this usually means that we can only add data, but never remove any fields from the system.

Note that in this case, we simplify the model as we send it out. The warrants for this inmate aren’t going out, and we just pull the latest status and release dates from the most up to date warrant. This is a good example of how we avoid exposing our internal state to the outside world, giving us the flexibility to change things later.

The question now is where does this data goes to? RavenDB ETL will write the data to an external database, and here we have a few options. First, we can define an ETL target for each of the known parties that want this data (each of the blocks and the Command & Control Center, at this time). But while that would work, it isn’t such a great idea. We’ll have to duplicate the ETL definition for each of those.

A better option is to send the (transformed) data to a dedicated database that will be our integration source. Consider the following example:


In this case, we can have this dedicated public database that exposes all the data that the Registration Office shares with the rest of the world. Any part that wants this information can setup external replication from this database to their own. In this manner, when the Intelligence Office decides to make it known that they also needs to access the inmate registration data, we can just add them as a replication destination for this database.

Another option is not to have each individual party in the prison share its own status, but have a single shared database that each of them write to. This can look like this:


In this case, any party that wants to share data will be writing it to the shared database, and anyone who reads it will have access to it through replication from there. This way, we define a data pipeline of all the shared data in the prison that anyone can hook up to.

This post is getting long enough that I’ll separate the discussion of the actual topology of the data and handling the incoming data to separate posts, stay tuned.

Properly getting into jailit’s not a crime to be in an invalid state

time to read 5 min | 806 words

imageWhen you need to write a form, one of the first things that pops to mind is what kind of validations are required. A good example would be the new user registration form:


This is such an ingrained instinct that we typically seek to stop such invalid states as soon as possible. If the business rules says that you cannot have a certain state, it is easiest to just ensure that you can never get into such a state.

If you are working in a system that exists purely within the scope of your application, that might actually be useful property to have. This is certainly something that you want in your code, because it means that you can make assumptions on invariants.

If you have a system that reflects the real world, however, you need to take into account several really annoying things:

  • Your model of the real world is not accurate, and will never be accurate.
  • The real world changes, sometimes quite rapidly and in a very annoying fashion.
  • Sometimes the invariant is real, but you gotta violate it anyway.
  • If the system doesn’t let the users do their job, they will do the job in spite of your system. That will lead to a system of workarounds. Eventually, you’ll have to support these workarounds. This may result in hair loss and significant amount of aggravation.

Consider the case that the prison in question is a minimum security prison, expected to have white collar inmates that have already been tried. Now you get an inmate that is accused of murder and is currently on trial. (Side note, prisons have a very different structure for people who still undergoing trial, because they aren’t convicted yet and because the amount of pressure that they are under is very high. They’ll typically be held in different facilities and treated differently from inmates that have already been convicted and tried).

What do you think will happen if the system refuses to accept such an inmate into the prison? Well, the guy is right there, and the prison commander has already authorized letting him in. So telling him that he should go back home because they “computer won’t let you in” is not going to be a workable solution.

Instead, we use a very different model for validation and acceptance of data. Instead of stopping the bad input as soon as possible, we raise a flag about it during processing, but we do allow the user to proceed, assuming they explicitly authorize this. It will look like this:


At the same time, we flag such records and require additional review of the data.

In most cases, by the way, the fact that the inmate is in residence is not something that can be ignored and will be in all reports on the state of the prison until the state changes (transferred, verdict given, etc).

This kind of thinking is important, because it means that the software intrinsically does not trust the data, but continue runs validation on it. This is a good practice to be in when dealing with systems that reflect the real world.

This does lead to an interesting question, where do we run these validations, and when?

The ground rules for a good application is that it is like an amoeba, it may have multiple locations that it accepts input, but that is the only way to push data in, through well defined channels.

imageThis is another way of saying that we don’t allow someone else to go and poke around in our database behind our back.

Any time that we accept new data, regardless of the channel it arrives in, we run it through the validation pipeline. And this validation pipeline can add (or mark as obsolete) validation issues that should be brought to the attention of the people in charge.

Note that these sort of validations are almost always very much business rules, not type issues. If someone’s birthday is in the future, you can feel free to very easily reject that data as purely invalid. But if someone’s release date is in the past, however, you might still need to accept them as an inmate (the paperwork are really in the mail, sometimes).

I think I’ll need another post just to talk about how to implement these validation rules and behaviors, but that is still down the line. The next post topic is going to be data flow between the different systems that we talked about so far.

Properly getting into jailPhysical architecture

time to read 6 min | 1058 words

imageWhen building a system for a prison you need to consider a few more levels of high availably than usual. In particular, you might need to operate while there are riots in progress, which may be prison wide, in a single block or section or in general while in a partial failure mode. Regardless of what is going on, you want maximum survivability for the system. If there is a riot in the next block, I still want to be able to do a proper count of the inmates in this block, for example. So each block must be able to operate independently from the others.  For that matter, the command & control center must always be operational and so does the medical ward, the registration (who is in the prison right now is something that is important to know especially when there are issues.

All of this means that we can’t rely on any centralized architecture and that we want maximum separation between the different pieces. The reason this post is titled physical architecture is just that, we are talking about physical machines that reside some distance from one another. Now, granted, prisons are, by their very nature, limited in size, so we don’t have to worry too much about the latency of calls. However, we are worried about other aspects. We have to assume low to no maintenance and the chance of riots / floods and possible locusts.

The best case scenario for us is that each server will be installed on its own block, with a UPS  if we are lucky. The whole thing residing in a locked cupboard in the block’s sergeant's office. Given typical conditions, the cupboard is also were the teapot and coffeemaker will reside, and we’ll have to worry about our server going doing because someone spilled a whole pot of tea on the machine.  But I’ll leave completely fictional heart attack inducing events to another day and focus on the software architecture.

We need to be able to operate each block independently of other blocks and from the central prison administration. That means that we will use an architecture that assumes that we are an island. All the applications that are used will only talk to a local database, and cannot assume any other connectivity being available.

Instead, we’ll build explicit data flow channels between the different parts of the system and handle that part explicitly.

It is important to note that this type of architecture also has another major benefit, it cushions certain type of errors. If an inmate managed to trip on a wire and cut the network cable leading to the block, we want to still be able to operate normally for all / most usual things. This means that we can summon the network guy to come and fix things in the morning, rather than have them come up now. That is the kind of architectural results that you like if you are one of the guys who might be called in such a case.

The block server will hold:

  • The inmates that are in the block and well as their record in the block (disciplinary actions, privileges withheld / given, notes, etc).
  • Records of counting of inmates.
  • Log of actions happening in the block (Sargent's diary).
  • Tasks and operations required (sending inmates to court, scheduled work parties, cell searches, etc)

That kind of data is essential for the routine operation of the block and should always be available.

At the prison’s level, you have all kind of departments and the kind of data they need to keep:

  • Medical – medical history and other medical stuff. This is a separate system, because there is nothing unique in a prison medical ward that require anything special other than the bars on the windows and better locks on the drug cabinets.
  • Registration – all the inmates in the prison as well as all their holding documents. These are the documents and authorization to withhold someone’s freedom, and it is stored and managed by this department. This is also the department that can generate work orders (send this inmate to this court) for the rest of the prison and is the primary outside contact.
  • Command & control center – has a view of everything and is in charge of tracking operations, but the only action it takes is to pass that information onward.

I’ll speak more about each of these departments at length in the future, but for now I want to focus on something that is very interesting. None of these departments actually need to talk to anyone else to do most of its job.

The C&C center needs to get reports from the blocks about counts, that inmates were sent to court, that the inmates that were supposed to be freed were actually freed, etc. But none of that has to happen through regular channels. In fact, in many cases, they’re multiple channels that are used on an ongoing basis.

In addition to having the block’s server notify the C&C systems that the counts were completed, the sergeant will also notify C&C directly via a phone call about the count and how many inmates are currently in residence in the block.

The prison software needs to record all of that. In fact, even though it looks like it is doing the same work twice, we need it to happen, for the same reason we have double entry book keeping. It helps catch issues and omissions.

So that leads to another architectural tenant for the system. Not only are the system made of physically separated components, but we are also considering each portion of the system as a consistency boundary.

I hesitated here with regards to the naming. I almost went with a security boundary, but that isn’t quite right. It isn’t that the other side is not trusted in the classic sense. They are trusted, but they aren’t verified. We’ll need to double check all such data as it goes into the system. But that leads to the next big challenge, how are we going to handle the flow of data and verification of the data across the system as a whole.

I think this post is long enough as it is, so I’ll keep that for the next one.

Properly getting into jailCounting Inmates and other hard problems

time to read 4 min | 788 words

imageIn prison, the notion of counting people is sacred. This is probably because the whole point of having a prison is to keep the people you put in there inside, and that means that you count inmates, multiple times a day.

The dirty secret, however, is that you almost never get to that perfect occupancy number, where all the inmates that are registered to a particular block are actually in that block. In most cases, you have at least a few that are outside the block (court dates, medical issues with offsite care, visitations, outside work, etc).

So the key here is not just to count the inmates, but to also account for them. Let’s consider how this will look like in the user interface for counting a couple of cells, shall we?


This works, but it isn’t a good idea. Named counts are usually reserved for the first / last counts of the day. During the day, because it is so frequent to have inmates in and out of their assigned location, you’ll usually do things differently. You’ll have a total count of inmates in the block, and a list of the exceptions. That would look something like this:


You have the number of inmates in the block, how many are expected to be there, the actual count as verified by the sergeant’s signature and the named list of inmates that are not currently in the block. You might have noticed that we are carefully tracking who is responsible for any inmate that it currently out of the block. This is because that matters (for a whole host of legal and cover your ass reasons).

So how would we build such a system?

To start with, we need to talk about the current component or service that we are building. The notion of counting is typically done at the block level, so we’ll start by modeling things there. We’ll have the Block Service, which is in charge of managing anything that is going on inside the block.

A block is composed of:

  • Cells, to which inmates are assigned. This is typically an internal division only that has no real business meaning.
  • Inmates, which are quite important.
  • Staff, which is probably a separate issue entirely, but is quite important for thing such as having enough people at hand to do things like actually run the block.

In terms of the actual operations we need to do, the block is managed by at least a single Sargent per shift and multiple guards. The Sargent is responsible for handling incoming inmates, counting all inmates multiple times a day and other things that we won’t be tracking on a computer system. The guards will mostly interact with the system when they need to check an inmate out for whatever reason (such as taking them to a checkup by a nurse).

With all of this information, we can now model the data we have for a block. Here is the most important document we have, the block’s population. There are a few things here that are worth exploring in the design of the document:


First, we have a separate document per date, recording the state of the block’s population at that time. This is important, because we need to be able to go back in time and track such things. You can also see that the document contains a lot of data that has both the name and id. Why is that?

The information recorded on this document is the data as it was at the time of this document’s creation. Later changes do no apply, by design, since we need to see keep it at the state it was at the time.  It might be easier to look at things in code:

The most important thing here is the notion of the Log, which records every incoming and outgoing inmate from the block.

In addition to the daily’s block population document, we also have three to five counting documents, which are built on top of it. These reflect the actual counts made plus the listing of inmates that aren’t currently in the block and why.

And that is quite enough about the prison’s life. This give us sufficient level of details that we can now work with. The next post will talk about how the physical architecture and data flow in the prison.

Properly getting into jailIntroduction & architecture

time to read 5 min | 808 words

imageI have been writing about the internals of RavenDB for quite a while, and it is fascinating in many respects, but it gets tiring to keep thinking about bits & bytes all the time. I’m missing writing some of the more high level stuff, in particular about software architecture. So I thought that I would take the time to resurrect a very old post series of mine, Macto. I actually have quite a few posts about this, but they are all close to a decade old, so I might as well start from scratch.

Macto is a sample app for managing a prison. This is one of the few areas in which I can be an expert on the business requirements that isn’t utterly boring (I don’t want to do yet another E-Commerce stuff).  To keep up with the times, Macto is going to be written with a Micro service architecture, and just for fun, I might be doing that in multiple languages and platforms, because prison is not fun, and neither should be working on it’s computing systems Smile. Oh, and there is also the real world thing, I guess.

Since 2009, I have pretty much given up on building anything UI wise, so I’m going to show a few mockup screens, but the idea is that I’m going to be looking only at backend code, with another team actually doing the user interface.

The first thing to start with, I guess, is to paint the overall architecture of the system. Prisons are pretty rigid systems, as you might expect, but there are a lot of interconnected parts. In order to properly build a system to manage a prison we need to be able to answer what is going on from multiple points of view and from very different perspectives.

The prison commander cares about such things as the inmate’s count (the job is to keep them all hale and accounted for). The individual guard care about the particular set of cells they are assigned to and the inmates in them. The guys at Registrations care about the legal details of having a lawful warrant for holding an inmate in prison as well as when an inmate should be released. Intelligence cares about the connections between inmates, the kind of heads up that came through channels and what kind of actions should be taken as a result. Medical needs to verify that incoming inmates are fit to be held in the facility and Transfers needs to ensure that any movements of inmates outside the prison will complete successfully (as in, you got them out, you also gotta bring them all in).

Each of those pieces interact with others in interesting and complex ways. For example, incoming inmates needs to go through Registration for legal paperwork, Medical for certification, Intelligence for verification and then assigned to the proper block.  Once they are accepted into the prison, they are the charge of the particular block they are assigned to and rarely need to interact with the rest of the prison unless something extraordinary happens (visits, court, sickness, etc).

When thinking about the software architecture of such a system, the most important rule to remember is that we want the system to be used, this means that we really need to plan for what people are actually doing (in abstract of what they should be doing) and to help them do things, rather than hinder them. In most places, all these details are done with pen & paper, and it works, so our system will have to offer something more. Not to the prison administration, but to the actual people going about their work with the inmates.

From a software architecture point of view, we are going to model the system as a set of independent services that will each have the role of one of the departments inside the prison. The current term in micro-services, but in real systems, they are not so micro, so we might need to repeatedly break things apart until we get to a level in which things make sense in isolation.

A lot of the complexity is involved in managing such a system is in the flow of information across the system. In a prison, this is the responsibility of the Command & Control Center (C3, from now on) which is in charge of coordination and monitoring of actions across the board. This also work very closely with the heads of all departments and the prison commander as well as most other external parties, but it generally does nothing on its own.

I think that this is enough of an intro, and we’ll get right on to things in the next post, where we’ll talk about Counting Inmates.


  1. RavenDB Security Review: Encrypting data on disk - 15 hours from now
  2. RavenDB Security Review: Encrypt, don’t obfuscate - 4 days from now

There are posts all the way to Mar 26, 2018


  1. RavenDB Security Review (4):
    22 Mar 2018 - Nonce reuse
  2. Inside RavenDB 4.0 (5):
    21 Mar 2018 - Chapters 12 & 13 are done
  3. Properly getting into jail (13):
    19 Mar 2018 - The almighty document
  4. Production postmortem (22):
    22 Feb 2018 - The unavailable Linux server
  5. Challenge (50):
    31 Jan 2018 - Find the bug in the fix–answer
View all series


Main feed Feed Stats
Comments feed   Comments Feed Stats