http://ayende.comAyende @ RahienCopyright (C) Ayende Rahien 2004 - 2021 (c) 202660All I can say is, yeah, right there with you. :-) http://ayende.com/4260/emergency-fixes-the-state-of-your-code-base#comment12http://ayende.com/4260/emergency-fixes-the-state-of-your-code-base#comment12Mon, 26 Oct 2009 14:06:15 GMTif I've learned one thing in doing 6 years of support on a software product it's that there are no bugs which need a fix so desperately you've to drop everything immediately. (well, except the small group of bugs which cause a corporate website go belly up). That's not to say there aren't severity levels, of course there are. But unless someone dies because of the bug, it's better to take a step back and take a little bit of extra time to come up with a better approach to test what could be wrong. Also, some customers of software products are simply over-reacting drama-queens, that's a fact of life: the issue is _so_ severe for them that everything in the world has to stop and everyone/thing has to put into gear to fix their problem. There's nothing wrong with that, it might be the issue is indeed that serious. But, unless the person is on the edge of losing his/her job because Big Corp Inc.'s website is showing a yellow screen of death, it's not that serious that you should go in head-first. You'll see that if you take 1-2 hours extra, take it easy and consistent, it will get the same result: fixing the bug, and not the mess. For a small ISV, it's hard to learn about a potential big bug which could potentially affect everyone: these things have to be fixed _now_ (it seems) as otherwise everyone who downloads the trial will likely give up with a "this is crap" remark. But that's not matching reality: these kind of severe issues are almost always already caught with your own testing, so severity is likely lowered by the fact these things are edge cases. Still not nice for people who run into these edge cases, but hardly a reason to drop everything. Because dropping everything to go in to fix an issue immediately not only causes mess around the bugfix, it also makes you lose extra time because you'll likely have to start over with the process you were working on (and which you dropped to fix the bug). The sloppy reader will now think I find people who ran into severe bugs should not get support at that moment. That's not true: every customer is king and should get support, as soon as possible. However, every customer also should take into account that the issue also might be their own fault and no-one wins with a head-first hasty 'fix'. http://ayende.com/4260/emergency-fixes-the-state-of-your-code-base#comment11http://ayende.com/4260/emergency-fixes-the-state-of-your-code-base#comment11Fri, 23 Oct 2009 08:51:01 GMTAlas, he is human ;) I call this *thrashing*. It's messy and horrible, but instinctively the best way to dig yourself out of a hole. http://ayende.com/4260/emergency-fixes-the-state-of-your-code-base#comment10http://ayende.com/4260/emergency-fixes-the-state-of-your-code-base#comment10Thu, 22 Oct 2009 21:36:10 GMTRafal, I full agree that this is maintenance/monitoring issues. http://ayende.com/4260/emergency-fixes-the-state-of-your-code-base#comment9http://ayende.com/4260/emergency-fixes-the-state-of-your-code-base#comment9Thu, 22 Oct 2009 21:34:46 GMTMy least favorite fixes are what I call "Panadol" fixes. (or "Aspirin") These are cases with data or state where the cause of the intermitent bug cannot be determined, but the symptom can, so the fix detects and fixes the result of the bug. Usually these are cases where I find a way to reproduce the problem, but when consulting with the client it isn't something they've done which means there's at least 1 more way to trigger the same buggy result. If I'm running out of time, I build the "Panadol" and make sure it does it's job, then fix the bug I know about. http://ayende.com/4260/emergency-fixes-the-state-of-your-code-base#comment8http://ayende.com/4260/emergency-fixes-the-state-of-your-code-base#comment8Thu, 22 Oct 2009 21:31:01 GMTAyende, enabling full logging at production server is not a bug, not even a programming issue. That's maintenance... If no one has noticed in 3 months time that disk is running out of space it means the system wasn't that important anyway. But I'm digressing. Regarding the quick hacks on production, I think many systems are riddled with them. Everyone speaks of them as quick&dirty temporary solution that will be soon replaced with the professional one and it stays there forever (especially after it gets sucked into code repository). Is it bad? I think not too bad - if it's still there after few years it means only that it has caused no problems and no one cares if looks nice or not. On the other hand quick fix on production saved my ass many times. That's why I like scripts so much and love Boo with Rhino.DSL - I can modify business logic on production in no time and go back to previous version if something goes bad. http://ayende.com/4260/emergency-fixes-the-state-of-your-code-base#comment7http://ayende.com/4260/emergency-fixes-the-state-of-your-code-base#comment7Thu, 22 Oct 2009 19:01:15 GMTI find it really maddening when programmers make speculative changes _to make the problem go away_ rather than to identify, isolate and fix the problem! After the 10th speculative change, the problem vanishes and is deemed "fixed", where in fact it's been moved somewhere else and the programmer has caused 3 other bugs! It's especially easy for people to fall into this trap when submitting one line fixes. After all, it's just one line; what could go wrong? :) http://ayende.com/4260/emergency-fixes-the-state-of-your-code-base#comment6http://ayende.com/4260/emergency-fixes-the-state-of-your-code-base#comment6Thu, 22 Oct 2009 16:26:37 GMTFor tracing problems in applications I use "Debugging Tools for Windows" more and more often. Instead of just guessing about the problem, I actually just manage to get a memory dump and later analyze that one. Works wonders in cases where exceptions only occur once in a while and noone has any idea how to reproduce them. Or when applications use up alot of CPU usage for no reason. Hangs. And the list goes on. http://ayende.com/4260/emergency-fixes-the-state-of-your-code-base#comment5http://ayende.com/4260/emergency-fixes-the-state-of-your-code-base#comment5Thu, 22 Oct 2009 15:37:59 GMTReminds me of this. [blog.hasmanythrough.com/2009/9/3/circle-of-death](http://blog.hasmanythrough.com/2009/9/3/circle-of-death)http://ayende.com/4260/emergency-fixes-the-state-of-your-code-base#comment4http://ayende.com/4260/emergency-fixes-the-state-of-your-code-base#comment4Thu, 22 Oct 2009 15:21:24 GMT...and crappy code is the #1 contributor to lack of sleep. http://ayende.com/4260/emergency-fixes-the-state-of-your-code-base#comment3http://ayende.com/4260/emergency-fixes-the-state-of-your-code-base#comment3Thu, 22 Oct 2009 14:57:31 GMTI found it to be very valuable to have an up to date copy of the production database so i can make experiments that potentially destroy data. http://ayende.com/4260/emergency-fixes-the-state-of-your-code-base#comment2http://ayende.com/4260/emergency-fixes-the-state-of-your-code-base#comment2Thu, 22 Oct 2009 14:00:34 GMTLack of sleep is the #1 contributor to crappy code. http://ayende.com/4260/emergency-fixes-the-state-of-your-code-base#comment1http://ayende.com/4260/emergency-fixes-the-state-of-your-code-base#comment1Thu, 22 Oct 2009 13:45:23 GMT