Build the tools that aren't there

time to read 8 min | 1445 words

This is strongly related to my posts about Tools Matter. During the last six months had several conversations with people about their Xyz processes. In several cases, my response was a polite version of: "This Is Broken, Badly" and "You should automate this part".

I am a great believer in automating just about everything that move (and if it doesn't move, kick it until it does!). In those conversation, the response was usually, "Yeah, we thought about doing that, but Abc does things in a way I don't like and Efg isn't compatible with our Foo requirements". And that was that.

Let us take deployment as good example of that. I was talking with someone about the need for automatic deployment, and he mentioned that he is waiting for a tool to come up that will also handle workflows.

I was a bit stunned by that, and inquired deeper, at which point it became clear that the guy was working in a highly regulated environment and doing a deployment involved multiple people authorizing it in different environments before it could go live. Because of all the manual work that already exists there, which cannot be changed for regulatory reasons, they have no automated deployment.

Note, it sounds much worse than it actually is, now that I re-read this.

I was critical on this approach, for several reasons. First, even if you can't go all the way, just having a build script that you have to manually run is a huge improvement. Next, I asked several additional questions bout the scenario, and it turned out that the process was something like this:

  • We need to push something to production
  • We deploy to a test server and ask QA to test that
  • Once QA sign off this release, deploy to a staging server
  • Get at least three business experts to smoke test the system
  • Once we have 3 signatures that authorize the system, we can ask Joe (the friendly IT admin) to deploy to production
  • Joe schedule a time for deployment and get signoff for that from someone with the authorization to do so.
  • At the specified time, Joe is going to deploy to production, the dev team is on call for issues there

This isn't a good going-to-production scenario, with multiple check points to ensure that we are safe & sound. The real reason is not to actually ensure quality, of course, it is to satisfy some dry regulation and have an audit trail that you can point to. But that is beside the point and shows my utter annoyance with all forms of bureaucracy.

Okay, so we have this process that we must go through in order to get something to production. There is no tool out there that will do it for us and give us the required audit trail. Therefor, we can't use automatic deployments.

My response for that was rude and unprintable.

Here is the deal, let us estimate the cost of building such a system:

  • A page into which I can enter request to go to production. This consists of two text boxes and a submit button. On submit:
    • Automatically deploy to the test server
    • Send an email to me and the QA department that we have something that they need to test
    • Record that I have started a deployment process
  • A page into which the QA department can say if they authorize the build or not. On submit:
    • If not:
      • record this fact
      • email to dev team
    • If yes
      • Record this fact
      • Automatically deploy to staging server
      • Email all the people that can approve a build and ask them to evaluate the build
  • A page into which the business experts can authorize or block a build. On submit:
    • If not:
      • record the reason
      • email to dev team and QA
    • If yes:
      • Record this fact
      • If this is the third person to authorize this build (and if there are no blocks):
        • record this fact
        • Send email to Joe, asking him to setup time for deployment
  • A page for Joe to enter time for going to production, on Submit:
    • record this fact
    • Send email to whoever it is that can authorize production downtime
    • Generate deployment package (which Joe will run in production)
  • A page for authorizing scheduled downtime:
    • Record this fact
    • Email Joe that the time is approved
    • Email whoever is interested that there will be scheduled downtime at that time

Five pages, more or less. And yes, I am glossing over things, I know. That is not the point.

If it takes over a week to build this I would be very surprised. The benefit is that we have a more streamlined process, we no longer have to babysit multiple manual deployments and Joe doesn't get some word document with instruction as to how to deploy. He gets a deployment package that he can copy to production and double click in order to deploy this.

Let us take another scenario. Deploying to production often fails because of one problem or the other, usually the IT admins who performed the install gave bad values to the build script. (Such as specifying the wrong connection string, or have a typo in some URL, or something of this nature). The second time that this has happened, it should be caught by the build script itself. The response I got when I expressed this opinion was that they had no control over the build process, that it was entirely the realm of the IT administrators.

Let us take the most difficult scenario that I can think of. We are required to hand to the IT admins the compiled binaries along with a document that specify what new values we should put in the configuration.

My approach for this would be to put an if statement in the application startup, which will perform a full environment check (the idea was stolen from Jeremy Miller and Release It!, by the way) and give a detailed error message. Since this is likely to be a long process, it will disable itself the first time it passes successfully (I leave the how as an exercise for the reader, consider that it should be reset the next time we deploy).

The tool to do that is code, your code. Which you built in order to provide you with the foundation for your project.

In short, because I have another half dozen examples that are as applicable, remember, you are a developer. If your tools doesn't provide you with what you need, you can build it. And since you are not going to try to build a generic tool, the cost of doing it is extremely low.

It doesn't even have to be a tool, just create a console application that does something, where you hard code everything. Let the compiler be your tool, and "configure" it with code.

Don't wait, act.