Don't start a demo with DDoS

time to read 2 min | 393 words

A few days ago I had a launch of my current project to the internal customers. This meant that many more users had a chance to see the nearly-complete application. There are still a lot to do, but we are very close to feature-freeze (yeah!).

Anyway, I closed down everything the day before (around noon, so I had plenty of time to verify that it worked), the day of the presentation, I took the liberty of arriving late (I often does, actually, it allows me to skip traffic). Around 20 minutes from work, I started to get urgent phones. We have an issue with this, and there is an error with that. The application is slow, there are timeout errors, etc.

I was absolutely bewildered, it worked yesterday, and no one has changed a thing. I think that I arrived to work about ten minutes before it was to go live to the users, and I quickly started the crisis mode maneuvers. Restart IIS, Restart SQL Server. It helped for a few minutes, but then the application started to show timeout errors from the database. I increased the command timeout (single line change, in the configuration, very cool) and the application seems to have stabilized, albeit very slow.

I then start looking more deeply at the root cause of the matter. One of the first places that I looked at was the requests/sec perf counter, and it was very high for a system that no one should be using at the moment. We got a fairly high numbers of requests per second (~17), sustained over a long period of time, it was as if I suddenly had a few hundreds very active users.

It was a good chance to brush off my forensic skills, and a short while after I have determined that the hits where coming from a small set of IPs, at a very high rate. Someone was DDoSing me. It took some exchange of blame and a whole round of denials before someone remembered that they are running the stress test kit against the demo servers.

The moment that stopped, everything went back to normal, but that guy is responsible for at least three separate breakdowns, a whole lot of swearing, and quite a bit of worrying.

 Not a good way to start a demo, but once we passed that, we have gotten very positive results. :-)