http://ayende.comAyende @ RahienCopyright (C) Ayende Rahien 2004 - 2021 (c) 202660You may want to check Windows Event Viewer around the time everything stopped working, perhaps it logged an error about the service that failed. http://ayende.com/4457/debugging-a-production-problem#comment13http://ayende.com/4457/debugging-a-production-problem#comment13Sun, 11 Apr 2010 03:08:43 GMTMike, Yes. I'll have a post about this shortly. http://ayende.com/4457/debugging-a-production-problem#comment12http://ayende.com/4457/debugging-a-production-problem#comment12Sat, 10 Apr 2010 09:02:32 GMTI seen this happen, during dev, when reading from a transactional MSMQ queue in a trasnaction that had elevated to MSDTC, I killed the process via task manager, and restart it, it would just hang. I had to restart the server on the occassions, not sure what MS DTC was trying to, or why it just didnt timeout. http://ayende.com/4457/debugging-a-production-problem#comment11http://ayende.com/4457/debugging-a-production-problem#comment11Sat, 10 Apr 2010 09:02:10 GMTNot directly related, but I'm curious why you don't use Rhino Queues? The tooling with msmq? http://ayende.com/4457/debugging-a-production-problem#comment10http://ayende.com/4457/debugging-a-production-problem#comment10Fri, 09 Apr 2010 19:03:45 GMTWhat I want to know is why the snipping tool doesn't have a horizontal highlight option. :( http://ayende.com/4457/debugging-a-production-problem#comment9http://ayende.com/4457/debugging-a-production-problem#comment9Fri, 09 Apr 2010 16:36:23 GMTCory, No, I didn't take a dump, if this will happen again, I certainly will. There wasn't any high CPU, and I am not sure what low CPU hang is http://ayende.com/4457/debugging-a-production-problem#comment8http://ayende.com/4457/debugging-a-production-problem#comment8Fri, 09 Apr 2010 16:23:08 GMTManaged Stack Explorer is a nice tool. Not as powerful as WinDbg, but faster for grabbing managed dumps and you don't have to restart the process: [http://www.codeplex.com/wikipage?ProjectName=MSE](http://www.codeplex.com/wikipage?ProjectName=MSE)http://ayende.com/4457/debugging-a-production-problem#comment7http://ayende.com/4457/debugging-a-production-problem#comment7Fri, 09 Apr 2010 13:57:20 GMTI work on a system that also uses MSMQ and receives messages in DTC-managed transactions. (We're writing to a database in the same transaction.) Occasionally our apps just stop processing messages even though they are still working in all other respects. WinDbg usually shows that we have a few threads stuck on ReceiveById(). Just like in your case, restaring DTC and/or MSMQ gets things going again. (Other details: We're using WCF's NetMsmqBinding and hosting the apps in IIS, so WAS and the Net.Msmq Listener Adapter are also involved.) Long way of saying: If you figure out what's happening, I'd love to hear about it! And if we figure it out I'll post the details here. http://ayende.com/4457/debugging-a-production-problem#comment6http://ayende.com/4457/debugging-a-production-problem#comment6Fri, 09 Apr 2010 13:29:28 GMTfor the powershell example, look at $error[0] | format-list * -for This will show you the full exception and stacktrace. James http://ayende.com/4457/debugging-a-production-problem#comment5http://ayende.com/4457/debugging-a-production-problem#comment5Fri, 09 Apr 2010 11:28:24 GMTUpdated the post :-) http://ayende.com/4457/debugging-a-production-problem#comment4http://ayende.com/4457/debugging-a-production-problem#comment4Fri, 09 Apr 2010 11:15:55 GMTI have had many problems with DTC, especially when there was an 'orphaned' transaction holding locks forever. This is one of things should never happen in theory but will certainly happen in production. http://ayende.com/4457/debugging-a-production-problem#comment3http://ayende.com/4457/debugging-a-production-problem#comment3Fri, 09 Apr 2010 10:29:29 GMTI'd love to know the end of that sentence, but it http://ayende.com/4457/debugging-a-production-problem#comment2http://ayende.com/4457/debugging-a-production-problem#comment2Fri, 09 Apr 2010 10:18:09 GMTbut it ... ? http://ayende.com/4457/debugging-a-production-problem#comment1http://ayende.com/4457/debugging-a-production-problem#comment1Fri, 09 Apr 2010 10:12:01 GMT