Ayende @ Rahien

My name is Oren Eini
Founder of Hibernating Rhinos LTD and RavenDB.
You can reach me by phone or email:


+972 52-548-6969

, @ Q c

Posts: 6,123 | Comments: 45,465

filter by tags archive

Sites outage

time to read 1 min | 182 words

We have an outage that appears to have taken roughly 12 hours.

The reason it took so long to fix, it was after business hours, and while we have production support for our clients, we never hooked up our own websites to our own system. A typical case of the barefoot shoemaker.

The reason for the outage? Also pretty typical:


The reason for that? We had a remote backup process that put some temp files and didn’t clean them up properly. The growth rate was about 3-6 MB a day, so no one really noticed.

The fix:


All is working now, I sorry for the delay in fixing this. We’ll be having some discussion here to see how we can avoid repeat issues like that.


Christian Seitzer

I would suggest a tool like nagios or one of its derivatives to monitor your hard disks.

My experience with icinga and nsclient++ for Windows has been very good..

Jiří Nouza

We use very simple powershell script to check server disk drive free space.

Jim Geurts

+1 for Nagios or Zabbix... you get lots of other built in metrics like cpu load, etc as well

Judah Gabriel Himango

Heheh, seen this plenty of times. Usually it's the IIS logs that hurt me.

Wyatt Barnett

Protip: if you are doing anything on your OS volume you are probably doing it wrong on a server. 1st setup step here is to move everything IIS related to D.


Ayende I am just happy you have 149GB of HibernatingRhinos.Orders :)


You should also probably set customErrors to On and set defaultRedirect to a nice error page that doesn't leak your stack trace...

Ayende Rahien

Robert, I don't do that on purpose.


When setting up servers I like to allocate a large file of several gigabytes that can be delete when this situation occurs. This has saved my bacon a few times when running out of space on source control repositories.

Daniel Marbach

Ayende I would like to introduce you to Oren Eini. He is the smart man behind RavenDB. In situations like that I always like to quote his excellence from his workshops: "disk space is cheap" :)

Comment preview

Comments have been closed on this topic.


  1. RavenDB 3.5 whirl wind tour: You want all the data, you can’t handle all the data - 3 days from now
  2. The design of RavenDB 4.0: Making Lucene reliable - 4 days from now
  3. RavenDB 3.5 whirl wind tour: I’ll find who is taking my I/O bandwidth and they SHALL pay - 5 days from now
  4. The design of RavenDB 4.0: Physically segregating collections - 6 days from now
  5. RavenDB 3.5 Whirlwind tour: I need to be free to explore my data - 7 days from now

And 13 more posts are pending...

There are posts all the way to May 30, 2016


  1. RavenDB 3.5 whirl wind tour (14):
    29 Apr 2016 - A large cluster goes into a bar and order N^2 drinks
  2. The design of RavenDB 4.0 (12):
    28 Apr 2016 - The implications of the blittable format
  3. Tasks for the new comer (2):
    15 Apr 2016 - Quartz.NET with RavenDB
  4. Code through the looking glass (5):
    18 Mar 2016 - And a linear search to rule them
  5. Find the bug (8):
    29 Feb 2016 - When you can't rely on your own identity
View all series



Main feed Feed Stats
Comments feed   Comments Feed Stats