Ayende @ Rahien

My name is Oren Eini
Founder of Hibernating Rhinos LTD and RavenDB.
You can reach me by phone or email:


+972 52-548-6969

, @ Q c

Posts: 6,125 | Comments: 45,492

filter by tags archive

Ideal deployment disaster

time to read 2 min | 301 words

Today we botched deployment for production. We discovered an unforeseen complication related to the production network topography which means that (to simplify) transactions wouldn’t work.

I am happy about this. Not because of the failed production deployment, but because of how it happened. We worked on deploying the app for half an hour, found the problem, then spent the next few hours trying to figure out the exact cause and options to fix it. We actually came up with four ways of resolving the issue, from the really hackish to what I believe that would be appropriate, but would take some time to do using the proper process for production changes.

So, why am I happy?

We spent time on that from noon to 3 PM (look Ma, no midnight deployment), and we actually aren’t going live any time soon. The reason for deploying for production was to practice, to see what strange new things would show up in production. They did show up, we will fix them, at our leisure, without trying to solve a hairy problem while staring at the screen with blood shot eyes and caffeine positioning.  At no time there was someone breathing down our neck, or the IT guys to argue with, or emergency change request or any of the usual crap that usually comes with deploying to production if you aren’t being careful.

Practice the way you play, and since pushing to production is usually such a mess anyway, you want to practice this as soon as possible. And not just deploying to another environment (we already did that last week) to test your deployment process. You want to deploy to the actual production environment, so you can get permissions, firewalls, subnets and all the rest sorted out up front.



that's why i encourage clients to have 3-4 environments: development, test, staging, and production. (not counting build machines) The idea of staging is to be a replica of production and is used for practicing deployments. Let's you shake out the loose parts, and get those deployment scripts right so the whole thing goes smoothly.

late night deployments can suck, but are a whole lot better when it only takes 20 minutes and everyone hangs out at the cafe down the block until Ops says its all clear. ..then its across the street to the bar for most.

Krzysztof Kozmic

That is all beautiful and true, but f rankly not always you CAN deploy your project to production before.

Grimace of Despair

"That is all beautiful and true, but f rankly not always you CAN deploy your project to production before."

I think that's part of his point. He has taken care of the possibility to deploy on a production environment.

Rick Kierner

Great idea. When I don't have access to the production machine, I like to make a VM Clone and setup that clone in the same network space and practice the deployment to the clone. It's not exact but it is better than a manual attempt to replicate the existing production machine.


So how do you deploy to production without affecting the existing production system?

Comment preview

Comments have been closed on this topic.


  1. RavenDB 3.5 Whirlwind tour: I need to be free to explore my data - 9 hours from now
  2. RavenDB 3.5 whirl wind tour: I'll have the 3+1 goodies to go, please - 3 days from now
  3. The design of RavenDB 4.0: Voron has a one track mind - 4 days from now
  4. RavenDB 3.5 whirl wind tour: Digging deep into the internals - 5 days from now
  5. The design of RavenDB 4.0: Separation of indexes and documents - 6 days from now

And 11 more posts are pending...

There are posts all the way to May 30, 2016


  1. The design of RavenDB 4.0 (14):
    05 May 2016 - Physically segregating collections
  2. RavenDB 3.5 whirl wind tour (14):
    04 May 2016 - I’ll find who is taking my I/O bandwidth and they SHALL pay
  3. Tasks for the new comer (2):
    15 Apr 2016 - Quartz.NET with RavenDB
  4. Code through the looking glass (5):
    18 Mar 2016 - And a linear search to rule them
  5. Find the bug (8):
    29 Feb 2016 - When you can't rely on your own identity
View all series


Main feed Feed Stats
Comments feed   Comments Feed Stats