http://ayende.comAyende @ RahienCopyright (C) Ayende Rahien 2004 - 2021 (c) 202660Something was striking me as odd about his assertions there, and even if you have the most stringent of testing, some environments (e.g. FDA regulated) still require you to place checks in your code to handle scenarios that "are never supposed to happen" http://ayende.com/4159/your-queries-are-going-to-fail-in-production#comment19http://ayende.com/4159/your-queries-are-going-to-fail-in-production#comment19Tue, 01 Sep 2009 14:50:32 GMTThe most common case of failing queries in production is IMHO a database update with feature change. This happens often especially with MySQL (even for minor version changes), but we've seen it with other DBMS too, e.g. PostgreSQL. http://ayende.com/4159/your-queries-are-going-to-fail-in-production#comment18http://ayende.com/4159/your-queries-are-going-to-fail-in-production#comment18Tue, 01 Sep 2009 13:37:18 GMTPaul, Yes, you have to. You can't just execute the same statements, there is logic that may be affected http://ayende.com/4159/your-queries-are-going-to-fail-in-production#comment17http://ayende.com/4159/your-queries-are-going-to-fail-in-production#comment17Tue, 01 Sep 2009 11:55:37 GMTAyende, when you mention "replaying the transaction" above, do you mean throw away the session and restart the whole unit of work from the beginning? http://ayende.com/4159/your-queries-are-going-to-fail-in-production#comment16http://ayende.com/4159/your-queries-are-going-to-fail-in-production#comment16Tue, 01 Sep 2009 11:50:41 GMTI did remember me... Not quite sure why I am forgotten every second time I have something useful to say. Perhaps the timeout is too low? Just checked, yes indeed just 1 month timeout on the cookie. http://ayende.com/4159/your-queries-are-going-to-fail-in-production#comment15http://ayende.com/4159/your-queries-are-going-to-fail-in-production#comment15Tue, 01 Sep 2009 09:14:34 GMTNo. They are a waste of time also... Last time they spent weeks finding out why our failover setup was failing. Just to have us find out that they dont really support failover unless it is in a domain setup. So in usual style, I implemented a workaround, which in this particular case actually ended up much cleaner. http://ayende.com/4159/your-queries-are-going-to-fail-in-production#comment14http://ayende.com/4159/your-queries-are-going-to-fail-in-production#comment14Tue, 01 Sep 2009 09:12:04 GMTThe remember me shouldn't be on a second time, you are already remember. IOW, it shouldn't forget you. And I fully agree about the problematic nature of silent aborts. Have you tried contacting PSS? http://ayende.com/4159/your-queries-are-going-to-fail-in-production#comment13http://ayende.com/4159/your-queries-are-going-to-fail-in-production#comment13Tue, 01 Sep 2009 09:08:11 GMTYes and No... Since the transaction is stopped without your knowledge and no error handling is done, then all subsequent writes are still written. Meaning that you cannot just assume that none of your work was completed, and can thus not repeat the transaction. btw. The "Remember me?" keeps forgetting to remember itself, thus forgetting me the second time a post is made. http://ayende.com/4159/your-queries-are-going-to-fail-in-production#comment12http://ayende.com/4159/your-queries-are-going-to-fail-in-production#comment12Tue, 01 Sep 2009 09:04:37 GMTInteresting, but while this is a bug, it doesn't relate to what I consider the most important thing about transactions: ACID You don't have data corruption here, you have a transaction that mysteriously dies (and rollback). As long as you can trust ACID, you can replay transactions http://ayende.com/4159/your-queries-are-going-to-fail-in-production#comment11http://ayende.com/4159/your-queries-are-going-to-fail-in-production#comment11Tue, 01 Sep 2009 09:01:04 GMT[connect.microsoft.com/.../ViewFeedback.aspx](https://connect.microsoft.com/SQLServer/feedback/ViewFeedback.aspx?FeedbackID=262291) I remember running into 1 more, but I dont think I reported that to M$ and the circumstances have now eluded me. http://ayende.com/4159/your-queries-are-going-to-fail-in-production#comment10http://ayende.com/4159/your-queries-are-going-to-fail-in-production#comment10Tue, 01 Sep 2009 08:50:46 GMTDennis, Do you have a repro that shows that transaction handling is broken on SQL Server? http://ayende.com/4159/your-queries-are-going-to-fail-in-production#comment9http://ayende.com/4159/your-queries-are-going-to-fail-in-production#comment9Tue, 01 Sep 2009 08:39:41 GMTWould be nice to just be able to replay the transaction. But SQL server has so many bugs related to its transaction handling, that you never quite know when your transaction is broken. And in usual style, they pretty much ignore whatever bug reports they get. http://ayende.com/4159/your-queries-are-going-to-fail-in-production#comment8http://ayende.com/4159/your-queries-are-going-to-fail-in-production#comment8Tue, 01 Sep 2009 08:37:32 GMTOr simply a mistake, of course. I hardly believe Davy isn't aware about this. http://ayende.com/4159/your-queries-are-going-to-fail-in-production#comment7http://ayende.com/4159/your-queries-are-going-to-fail-in-production#comment7Tue, 01 Sep 2009 08:24:08 GMTFully agree. Failing query in production is normal. Deadlocks, version conflicts & timeouts are typical cases. Their ignorance = lack of actual understanding of processes @ RDBMS. http://ayende.com/4159/your-queries-are-going-to-fail-in-production#comment6http://ayende.com/4159/your-queries-are-going-to-fail-in-production#comment6Tue, 01 Sep 2009 08:14:50 GMTreplaying the transaction http://ayende.com/4159/your-queries-are-going-to-fail-in-production#comment5http://ayende.com/4159/your-queries-are-going-to-fail-in-production#comment5Mon, 31 Aug 2009 11:37:09 GMTAyende, can you elaborate on recovering aspects? Timeouts ??? http://ayende.com/4159/your-queries-are-going-to-fail-in-production#comment4http://ayende.com/4159/your-queries-are-going-to-fail-in-production#comment4Mon, 31 Aug 2009 11:35:17 GMTAll true. It also makes me think about the common assumption most developers have about Transaction Commits that generally don't fail. There are also a number of cases where this can happen (e.g. deferrable constraints in Oracle) but is usually not well handled. Basically, to develop robust software, you must be paranoid :) (re-re-read M. Nygard's excellent Release It !). http://ayende.com/4159/your-queries-are-going-to-fail-in-production#comment3http://ayende.com/4159/your-queries-are-going-to-fail-in-production#comment3Mon, 31 Aug 2009 11:06:35 GMTNima, There aren't really ways of getting rid of deadlocks. They are what happens when you have two transactions competing for the same resources. In theory, you can try to restructure your queries so you will always access rows using the same path all the time, thus eliminating deadlocks. In practice, it is not really possible to do something like that. You can reduce deadlocks by specifying a lower transaction isolation level, but preventing them isn't likely in real world scenarios. Luckily, recovering from them is very simple http://ayende.com/4159/your-queries-are-going-to-fail-in-production#comment2http://ayende.com/4159/your-queries-are-going-to-fail-in-production#comment2Mon, 31 Aug 2009 09:09:11 GMTIt might be irrelevant to your post but since you have mentioned deadlocks , I really like to know what's your advise (as a NHibernate professional) to prevent them ? As a matter of fact I've been working on a projects for 4 years and we are using NHibernate. In rare cases we are getting deadlocks .we have tried to trace them in SQL server logs but we haven't had any success to get rid of them. http://ayende.com/4159/your-queries-are-going-to-fail-in-production#comment1http://ayende.com/4159/your-queries-are-going-to-fail-in-production#comment1Mon, 31 Aug 2009 08:51:00 GMT