http://ayende.comAyende @ RahienCopyright (C) Ayende Rahien 2004 - 2021 (c) 202660Good reason to make sure you always have money available from multiple banks, and carry some cash -- although whenever I carry cash, I spend it :-) http://ayende.com/3719/not-a-production-quality-software#comment13http://ayende.com/3719/not-a-production-quality-software#comment13Thu, 27 Nov 2008 00:48:15 GMTIt seems to me that the larger and organization, and certainly the more critical a system the less willingness there is to refactor code. It's a case of don't fix it if it ain't broken. So a league of programmers over many years do nothing but change the one little piece they were instructed to change. Very little attention is paid to looking at the behavior of the code and realizing this could be refactored into something shared, etc. because that might break things. This goes on for several years until you have a big mess nobody can understand, and everybody fears to change because they don't know what they might break. Usually at this point some new technology comes along and everybody advocates it will solve all their problems and they should rewrite. http://ayende.com/3719/not-a-production-quality-software#comment12http://ayende.com/3719/not-a-production-quality-software#comment12Tue, 25 Nov 2008 18:45:37 GMTI'd like to suggest that you rephrase your observation, and name it Ayende's Law. My recommendation for the new phrasing would be: "The quality of a software system is inversely proportional to the amount of money it handles." I'd also like to offer Xenolinguist's Corollary to Ayende's Law: "The quality of a software system is inversely proportional to the size of the company owning it when it was written." http://ayende.com/3719/not-a-production-quality-software#comment11http://ayende.com/3719/not-a-production-quality-software#comment11Tue, 25 Nov 2008 17:31:38 GMTHow many lines of code does it take to stuff money under the mattress? Now I am worried that US banks might have the same problem. http://ayende.com/3719/not-a-production-quality-software#comment10http://ayende.com/3719/not-a-production-quality-software#comment10Tue, 25 Nov 2008 16:52:16 GMTMaybe that's because I haven't had any transactions in the last few days :) http://ayende.com/3719/not-a-production-quality-software#comment9http://ayende.com/3719/not-a-production-quality-software#comment9Tue, 25 Nov 2008 12:22:09 GMTconfigurator, To the best of my knowledge, the reason was system upgrade. No mention of flooding. Also, when I was at the bank, at my own branch, they weren't able to tell me my current balance, only what it was several days ago, which was grossly inaccurate. http://ayende.com/3719/not-a-production-quality-software#comment8http://ayende.com/3719/not-a-production-quality-software#comment8Tue, 25 Nov 2008 12:04:15 GMTSome credit cards worked, mind you; the ones with less total purchases per than month the limit your credit company (MasterCard, Visa, etc.) explicitly set when you got the credit card do not need bank confirmation; luckily, I have a card I don't normally use so I was way under that limit. Also, money could be withdrawn and transactions could be made by going to the bank, which was open (relatively) late. I'd like to point that as far as I know, all transactions are printed as soon as they occur. I've seen a folder in my bank that had every bank transaction; not credit card transactions, as they are with the credit company, but they could know my exact balance without any computer aid; that much is a little comforting. And now I'll stop my worthless jabber and say what I wanted to say: as I heard it, the problem was because of a flood in the server room. I could be dead wrong - but do you have any evidence or trustworthy leads to the contrary? http://ayende.com/3719/not-a-production-quality-software#comment7http://ayende.com/3719/not-a-production-quality-software#comment7Tue, 25 Nov 2008 11:48:32 GMTAs more and more aspects of our lives are moved to data centers, we become more vulnerable to various errors or criminal activity. The disaster recovery is very hard, because verifying and fixing huge amounts of data after a crash is almost beyond human ability. If no one knows what the business rules are, how are they supposed to tell whether data is corrupt or not? I'm afraid the complexity and amount of data processed each day will some day make people slaves to technology - no human will be able to understand or reverse-engineer all the systems and virtual information will become the only reality - I mean the truth will be what the database says, not the opposite. This started quite innocently - with Google - remember? "If it's not in Google, it doesn't exist". Yes, I know it sounds like matrix intro, but I don't want to live at some corporation's mercy... http://ayende.com/3719/not-a-production-quality-software#comment6http://ayende.com/3719/not-a-production-quality-software#comment6Tue, 25 Nov 2008 11:22:47 GMTI can confirm your observation. In a project, I've seen a class of 5000 lines of pure business logic used to quote shares in real-time. It was literaly dealing with hundreds of millions euros every day and had exactly 0 test. It's also worth mentioning that an error of a few cents could instantly result in a loss of a few millions euros. The other interesting side of it is that nobody in the team was able to claim what the business rules were anymore. They were just there and that's it! Good stuff! http://ayende.com/3719/not-a-production-quality-software#comment5http://ayende.com/3719/not-a-production-quality-software#comment5Tue, 25 Nov 2008 09:16:41 GMTWhat you don't know can't hurt you eh? I had the exact same experience, except I went to work for an airline. Boy, that was scary! I have opted to remain anonymous :-) http://ayende.com/3719/not-a-production-quality-software#comment4http://ayende.com/3719/not-a-production-quality-software#comment4Tue, 25 Nov 2008 08:33:40 GMT>I am in no way naive enough to think that the situation is different in other banks, but at least I didn't know how bad it was. Just like with restaurants :) http://ayende.com/3719/not-a-production-quality-software#comment3http://ayende.com/3719/not-a-production-quality-software#comment3Tue, 25 Nov 2008 08:24:51 GMTUm. What paperwork? Does anybody actually have any paperwork about customer's accounts these days? Especially transaction-related stuff? Since Oren does not seem to complain about any of his money suddenly disappearing, I guess the bank did not actually loose the database. I guess 60 hours was basically them working around the clock to reinstall lots of stuff from scratch. Why didn't they have a DR site they can switch to when primary one becomes messed up is a good question. http://ayende.com/3719/not-a-production-quality-software#comment2http://ayende.com/3719/not-a-production-quality-software#comment2Tue, 25 Nov 2008 08:16:09 GMTI guess the disaster recovery procedure is something like: "collect all paperwork about this customer and try to reconstruct his/her account" http://ayende.com/3719/not-a-production-quality-software#comment1http://ayende.com/3719/not-a-production-quality-software#comment1Tue, 25 Nov 2008 07:45:52 GMT