Build the tools that aren't there
This is strongly related to my posts about Tools Matter. During the last six months had several conversations with people about their Xyz processes. In several cases, my response was a polite version of: "This Is Broken, Badly" and "You should automate this part".
I am a great believer in automating just about everything that move (and if it doesn't move, kick it until it does!). In those conversation, the response was usually, "Yeah, we thought about doing that, but Abc does things in a way I don't like and Efg isn't compatible with our Foo requirements". And that was that.
Let us take deployment as good example of that. I was talking with someone about the need for automatic deployment, and he mentioned that he is waiting for a tool to come up that will also handle workflows.
I was a bit stunned by that, and inquired deeper, at which point it became clear that the guy was working in a highly regulated environment and doing a deployment involved multiple people authorizing it in different environments before it could go live. Because of all the manual work that already exists there, which cannot be changed for regulatory reasons, they have no automated deployment.
Note, it sounds much worse than it actually is, now that I re-read this.
I was critical on this approach, for several reasons. First, even if you can't go all the way, just having a build script that you have to manually run is a huge improvement. Next, I asked several additional questions bout the scenario, and it turned out that the process was something like this:
- We need to push something to production
- We deploy to a test server and ask QA to test that
- Once QA sign off this release, deploy to a staging server
- Get at least three business experts to smoke test the system
- Once we have 3 signatures that authorize the system, we can ask Joe (the friendly IT admin) to deploy to production
- Joe schedule a time for deployment and get signoff for that from someone with the authorization to do so.
- At the specified time, Joe is going to deploy to production, the dev team is on call for issues there
This isn't a good going-to-production scenario, with multiple check points to ensure that we are safe & sound. The real reason is not to actually ensure quality, of course, it is to satisfy some dry regulation and have an audit trail that you can point to. But that is beside the point and shows my utter annoyance with all forms of bureaucracy.
Okay, so we have this process that we must go through in order to get something to production. There is no tool out there that will do it for us and give us the required audit trail. Therefor, we can't use automatic deployments.
My response for that was rude and unprintable.
Here is the deal, let us estimate the cost of building such a system:
- A page into which I can enter request to go to production. This consists of two text boxes and a submit button. On submit:
- Automatically deploy to the test server
- Send an email to me and the QA department that we have something that they need to test
- Record that I have started a deployment process
- A page into which the QA department can say if they authorize the build or not. On submit:
- If not:
- record this fact
- email to dev team
- If yes
- Record this fact
- Automatically deploy to staging server
- Email all the people that can approve a build and ask them to evaluate the build
- A page into which the business experts can authorize or block a build. On submit:
- If not:
- record the reason
- email to dev team and QA
- If yes:
- Record this fact
- If this is the third person to authorize this build (and if there are no blocks):
- record this fact
- Send email to Joe, asking him to setup time for deployment
- A page for Joe to enter time for going to production, on Submit:
- record this fact
- Send email to whoever it is that can authorize production downtime
- Generate deployment package (which Joe will run in production)
- A page for authorizing scheduled downtime:
- Record this fact
- Email Joe that the time is approved
- Email whoever is interested that there will be scheduled downtime at that time
Five pages, more or less. And yes, I am glossing over things, I know. That is not the point.
If it takes over a week to build this I would be very surprised. The benefit is that we have a more streamlined process, we no longer have to babysit multiple manual deployments and Joe doesn't get some word document with instruction as to how to deploy. He gets a deployment package that he can copy to production and double click in order to deploy this.
Let us take another scenario. Deploying to production often fails because of one problem or the other, usually the IT admins who performed the install gave bad values to the build script. (Such as specifying the wrong connection string, or have a typo in some URL, or something of this nature). The second time that this has happened, it should be caught by the build script itself. The response I got when I expressed this opinion was that they had no control over the build process, that it was entirely the realm of the IT administrators.
Let us take the most difficult scenario that I can think of. We are required to hand to the IT admins the compiled binaries along with a document that specify what new values we should put in the configuration.
My approach for this would be to put an if statement in the application startup, which will perform a full environment check (the idea was stolen from Jeremy Miller and Release It!, by the way) and give a detailed error message. Since this is likely to be a long process, it will disable itself the first time it passes successfully (I leave the how as an exercise for the reader, consider that it should be reset the next time we deploy).
The tool to do that is code, your code. Which you built in order to provide you with the foundation for your project.
In short, because I have another half dozen examples that are as applicable, remember, you are a developer. If your tools doesn't provide you with what you need, you can build it. And since you are not going to try to build a generic tool, the cost of doing it is extremely low.
It doesn't even have to be a tool, just create a console application that does something, where you hard code everything. Let the compiler be your tool, and "configure" it with code.
Don't wait, act.
Comments
It's amazing how much money organizations will spend building a software application, only to spend ridiculous amounts of money to manually pass it through their convoluted and backwards human processes.
It turns out that computers are actually pretty fast at these things and if you just tell it what your process is, it'll work!
I suggested to one guy awhile back that, to keep things simple, have an NUnit suite that checks for 'approvals' at various points along the sign-off chain. When you're ready to move from Dev to Staging or Staging to Prod, have the tests check for the necessary approval files (by name or content, whatever). Integrate the process RIGHT IN THE CODE!
He didn't like that idea and thought it was better done manually. Any automation is better than no automation, IMHO.
Oren I agree with you from a technical stand point. Now if you asked me the same question you asked that guy I can give you a straight face answer "Why bother?"
This is for two reason, first either way you'll get pay... I might even have some sort of automation in the background. But as far as the bureaucracy they don't exist :). This keep them happy and off your back and give you more time to goof around perhaps even time to take on a side project.
This might sound unethical but if some people actually will faults you for doing what's good for them so why bother. Assume that it take only a week to complete the automation process it's another week for them to get you on the hot seat since this isn't something that the bureaucracy dreamed of. Secondly if something goes wrong with the automation process it's something else for them to blame. Sometimes it's amazing what kind of idiot you have to deal with in the bureaucracy. It become senseless to try to beat any sense into them. So either just go with the flow or quit the job (I chose the later eventually)
It only takes a single botched deployment to show you why you really want to automate the entire process.
And yes, I would consider such behavior annoying, frustrating and unethical
What you're saying is right many more times than it's wrong, but the cost of maintaining these scripts isn't zero. If your workflow is changing too fast (disorganised or just agile, you decide!) it can become prohibitive. But nearly everything that can be automated should.
Greg,
Actually, no.
Check my JFHCI posts for more details
Oren, oh I agree with you entirely. It's annoying and frustrating. It's easy to persuade another developer to buy in on the automation process. Doing that to the management on the other hand doesn't always work. I am not saying it's not worth trying but sometimes it even have the reverse affect. If something goes wrong they come back and blame it on you for something else that's completely irrelevance. Of course you can always have the option to quit the job which is what I did. I am just saying from my person experience it's not always worth while to beat some sense into some senseless people. Sometimes they like it better for you to take advantage of them. It's crazy :)
firefly,
Management has no more say into automation than they have about the number of spaces I use to separate operators.
Literally, not their business.
"Management has no more say into automation than they have about the number of spaces I use to separate operators. Literally, not their business. "
Unfortunately, there are organizations where that simply isn't true. Devs have a lot of leeway in automating things, but 'Ops' can veto anything if they choose (for instance).
It is preferable, of course, to give them something to veto as opposed to just whining about the difficulties of life. They don't always (and if you can get them to ask for some task to be automated and then give them 10 times what they asked for, all the better).
jdn,
I gave just such scenario in the actual post.
Not a real problem
Hmm, isn't your answer to the scenario in your post the 5-page thingy?
I've worked in environments where they did just that, and it worked quite well (or, well enough). But in other situations, it's a non-starter, you couldn't really even suggest such a thing (and not only for bad reasons....when a deployment to production takes at least 12 hours, there's a lot that is required to get approval/sign-off, as you know).
It is a real problem.
The importance of automating everything you can cannot be stressed enough. Even for "one man Open Source shows" this will increase the developer's believe in that he can fast create a build without spending a complete afternoon (or week or month) which means that he will in fact create builds more often and thereby fulfilling the "release early, release often" mantra...
I have been working (as a consultancy) at places where I would basically quit if I were hired on a regular basis due to the complex build and deploy cycle they had.
If nAnt can't do it then break it till it does ;)
Jdn,
What 5 page thingy? The app?
That was for managing the audit trail. You were talking about letting the business dictate how it does to production, which is not acceptable.
And if going to production takes 12 hours, you are doing too much things manually.
"letting the business dictate how it (g)oes to production, which is not acceptable."
It's...highly sub-optimal. I'll certainly grant you that.
Bureaucracy is made one mistake at a time as a reaction to each incident which is corrected for
http://ferventcoder.com/archive/2008/09/07/tools-matter-automated-builds--automated-deployments.aspx
Strange. We were thinking of the same thing. :D
Oren, your post just trigger a memory :) That's all. It shouldn't be their business but in some place it is. I am all for automation myself that's one of the reason I become a programmer in the first place.
Comment preview