Ayende @ Rahien

Lothan commented on What am I missing? MSMQ Perf Issue

Thu, 22 Oct 2009 05:02:01 GMT

It's definitely the single-item transaction, Ayende. I copied your initial code and then modified it to perform two runs. The first run without transactions: SendData: 269 milliseconds CopyData: 554 milliseconds I then wrapped the loops in a transaction, so I'm committing all 10,000 messages in a single transaction: SendData: 481 milliseconds CopyData: 1,944 milliseconds Your original code that commits each item in a separate transaction: SendData: 16,107 milliseconds CopyData: 194,201 milliseconds So, yeah, the transaction cost here is astronomical and seems to have some sort of exponential function.

Ayende Rahien commented on What am I missing? MSMQ Perf Issue

Wed, 21 Oct 2009 20:29:19 GMT

Alois, This is using a stream to do stuff, not a serializer.

Alois Kraus commented on What am I missing? MSMQ Perf Issue

Wed, 21 Oct 2009 20:09:07 GMT

One very obvious thing: With what serializer do you create your msmq messages? The default (xml) is by far the slowest one. You should set it explicitely to get most out of it. Yours, Alois Kraus

David commented on What am I missing? MSMQ Perf Issue

Tue, 20 Oct 2009 16:54:39 GMT

I don't know if it's true with MSMQ 4.0, but with earlier versions, having journaling enabled significantly increases overhead when receiveing messages from a queue.

Ayende Rahien commented on What am I missing? MSMQ Perf Issue

Tue, 20 Oct 2009 10:02:10 GMT

Steve, Yes, I'll probably have to go this way, because of the horrible perf of tx in MSMQ. Very disappointed, thanks for the ideas

Steve Py commented on What am I missing? MSMQ Perf Issue

Tue, 20 Oct 2009 01:46:52 GMT

Yeah, in that scenario I would advise having the sender manage contingency for it's message not being received and processed rather than transactions. I don't seen anything with what you've done that might suggest why the transaction is more expensive than it should be. (Other than maybe disk latency as Gerke mentioned.) Services where I use MSMQ always have their own ACK private queue available for acknowledgements / status updates of their respective messages that they pass the address along as part of the request message. If they don't get an ACK to their satisfaction, they ping the queue for their request (in case it's still pending) and re-send if they don't find it. It's simple to wrap in, and if the scenario only manifests itself <2% of the time, it's a lot better option than accepting a 15-50% (or larger) performance hit overall. Of course if you're looking for "fire & forget" where the poster can't be assumed to be responsible for checking on its own requests, then either accept the performance hit of the transaciton, or consider using a database instead. Where I would use transactions is if I needed to ensure a message makes it onto multiple queues successfully, or situations where messaging is used to transmit sequential messages or messages on demand between 2 or more points. Chain is satisfied, commit. Ah well, good luck in any case. ;)

Jesse Ezell commented on What am I missing? MSMQ Perf Issue

Tue, 20 Oct 2009 00:52:53 GMT

Here's how to do it with WCF. [msdn.microsoft.com/en-us/library/aa395219.aspx](http://msdn.microsoft.com/en-us/library/aa395219.aspx) You can do the same type of thing manually with MSMQ. The big slowdown appears to be due to the transactions themselves, so if you can send multiple messages in a single transaction, you will get significantly better performance.

Gerke Geurts commented on What am I missing? MSMQ Perf Issue

Mon, 19 Oct 2009 23:59:31 GMT

I remember similar perf issues a long time ago with MSMQ transactions that were caused by slow write caching of the RAID controller. Replacement of the RAID controller improved a lot. Maybe MSMQ transactions are rather sensitive to disk write latency.

Ayende Rahien commented on What am I missing? MSMQ Perf Issue

Mon, 19 Oct 2009 23:48:27 GMT

Jesse, Can you give me a link to this, I am not familiar with the term

Ayende Rahien commented on What am I missing? MSMQ Perf Issue

Mon, 19 Oct 2009 23:47:58 GMT

Steve, Consider the simple case of: tx.begin order_msg = read_order_from_queue if process_msg order_msg: send_msg_to_warehouse ship_order tx.commit I have a lot of logic that does this. If I crash, I want things to recover properly. If I need to manage stuff manually, I might as well drop transactions all together.

Steve Py commented on What am I missing? MSMQ Perf Issue

Mon, 19 Oct 2009 22:23:20 GMT

I've never had to try and use transactions in this manner. You're likely going to get nailed trying to commit a change to a Queue that is actively receiving messages. My suggestion around Peek was if the _order_ was significant which would imply _1_ listener. As for losing messages, that is an exceptional circumstance. Could it be handled by the poster of the message if it's expecting a response? MSMQ exposes quite a bit of functionality such as PeekById and prioritization to help detect and re-send dropped messages. You may want to reconsider whether the cost of attempting to guarantee delivery is worth it, or just handle the exceptional case. (ala using UDP vs TCP) I don't know if this will work, but have you tried moving the creation of the transaction instance itself outside of the loop? (Still do the Begin and Commit inside the loop for each message, but avoid recreating a new transaction instance each iteration.) Not sure if transactions can be re-used like this or not...

Jesse Ezell commented on What am I missing? MSMQ Perf Issue

Mon, 19 Oct 2009 22:22:28 GMT

Yes. MSMQ transactions are EXTREMELY slow. I have seen similar behavior. To get decent numbers with transactions, you must use something like transaction batching. Of course, if you just switch to WCF and net.msmq binding, this is provided out of the box.

Ayende Rahien commented on What am I missing? MSMQ Perf Issue

Mon, 19 Oct 2009 18:52:19 GMT

Kerry, No, this is just test code. I am trying to see why my app (which does something similar) acts this way

Kerry Jenkins commented on What am I missing? MSMQ Perf Issue

Mon, 19 Oct 2009 18:45:03 GMT

Would it meet your requirements to create one transaction for all of the moves? I tried this by moving the "while (true)" statement down to after the msmqTx.Begin and moved the msmqTx.Commit after the ending "while" brace. This dramatically sped up the processing. I tested what an abort would do where the commit would normally happen after 10000 records were moved. It was very fast as well.

Ayende Rahien commented on What am I missing? MSMQ Perf Issue

Mon, 19 Oct 2009 11:11:20 GMT

Rafal, Yes, that is why I am shocked by this.

Rafal commented on What am I missing? MSMQ Perf Issue

Mon, 19 Oct 2009 10:10:53 GMT

Ayende, thanks for pointing this out, it's worse than I thought. But there's a post on Udi Dahan's blog, regarding NServiceBus performance ( [www.udidahan.com/.../nservicebus-performance/](http://www.udidahan.com/2008/05/21/nservicebus-performance/)) and he says: "OK, so using the default nServiceBus distribution using MSMQ, on servers where the queue files themselves were on separate SCSI RAID disks, we were pumping around 1000 durable, transactionally processed messages per second, per server."

Ayende Rahien commented on What am I missing? MSMQ Perf Issue

Mon, 19 Oct 2009 08:05:49 GMT

Rafal, In the test code, I AM using local MSMQ transaction

Ayende Rahien commented on What am I missing? MSMQ Perf Issue

Mon, 19 Oct 2009 08:05:39 GMT

Steve, What happens if my process crashes? A message is lost And Peek will let two concurrent consumers read the same message (BAD!) I am also trying to coordinate between two queues, so I _want_ to make use of this.

Rafal commented on What am I missing? MSMQ Perf Issue

Mon, 19 Oct 2009 07:55:47 GMT

@Steve - you probably are right, there's no need to enclose everything in a distributed transaction if the only thing you have to do is to put message back into the queue. You could use 'local' msmq transaction for that purpose and a separate, distributed transaction for the message handler.

Steve Py commented on What am I missing? MSMQ Perf Issue

Sun, 18 Oct 2009 23:20:29 GMT

I think the trouble is that your transaction is guarding an operation against a queue that is also receiving messages. The premise is that if something fails, the record from the receive transaction will be "put back", and this might be causing your performance headache. From your example I don't believe a transaction is required. If the order of the received messages is important then use Peek, attempt to process, and if successful (or determined to be poisoned) pull it off the queue (I.e. ReceiveByID). If the order of messages isn't important read it, attempt to process, then if it fails, decide whether to pop it back on to try again, or discard. Transactions carry a pretty steep performance hit in MSMQ and are really geared towards coordinating multiple queues.

Ayende Rahien commented on What am I missing? MSMQ Perf Issue

Sun, 18 Oct 2009 22:19:18 GMT

J Healy, Yes, RSB works with RQ

J Healy commented on What am I missing? MSMQ Perf Issue

Sun, 18 Oct 2009 22:06:21 GMT

Ah, that's right - I've clearly been distracted over the summer and need to catch up with things a bit. As I [vaguely] recall you were going to get Rhino SB running on top of your Queues, did that happen?

Ayende Rahien commented on What am I missing? MSMQ Perf Issue

Sun, 18 Oct 2009 21:40:18 GMT

J Healy, You are aware that there IS Rhino Queues project, right?

J Healy commented on What am I missing? MSMQ Perf Issue

Sun, 18 Oct 2009 21:37:48 GMT

I'm guessing RhinoMQ and RhinoTX will be in extended beta sometime tomorrow afternoon...

Ayende Rahien commented on What am I missing? MSMQ Perf Issue

Sun, 18 Oct 2009 20:01:28 GMT

Rafal, Yes, this affects RSB, because RSB make use of MSMQ. I still need to test how RQ behaves here.

Rafal commented on What am I missing? MSMQ Perf Issue

Sun, 18 Oct 2009 18:18:10 GMT

Wow, quite shocking. I'm using custom message queuing implementation based on a table in SQL server database and I'm getting about 150 messages per second (while enclosing message handler in a TransactionScope). And I was wondering if moving to msmq will allow me to handle thousands messages per second - looks like i wouldn't even be able to have current performance. @Frank - the problem is that you rarely can send or receive messages in batches, usually each message is an individual transaction. @Ayende - I suspect this also affects RSB performance as RSB is using distributed transactions?

Frank Quednau commented on What am I missing? MSMQ Perf Issue

Sun, 18 Oct 2009 16:22:26 GMT

Just for comparison, on my machine, 32-bit Dual core, 3GB RAM it takes 162 seconds. Not superfast, but about a minute faster than your measurement. This is on Vista...haven't used MSMQ for a while so I had to do a fresh MSMQ feature install. Installed without any addons like triggers HTTP support, etc. - just the core. Reducing the commit load by always reading and then writing 10 messages in a single transaction literally slahed the time to 1/10th, showing that the commit indeed is by far the most costly operation in the code.

Ayende Rahien commented on What am I missing? MSMQ Perf Issue

Sun, 18 Oct 2009 15:07:29 GMT

Chris, We are seeing something similar on Win2003 Server as well, though

Chris Patterson commented on What am I missing? MSMQ Perf Issue

Sun, 18 Oct 2009 15:05:38 GMT

I noticed something very similar Ayende. With Windows Server 2003 using .Single I would easily get 2k messages a second with load tests. On Server 2008/w7 I get on the order of maybe 350 pt second. Running it under dotTrace, all of the time is spent in MSMQ (my total application touch time is less than 500ms compared to like 22 seconds in MSMQ). I figure it had to be something to do with the transaction support on 2k8/vista/7 compared to 2003 server. But you're right, it's dog slow on the newer OS.

Ayende Rahien commented on What am I missing? MSMQ Perf Issue

Sun, 18 Oct 2009 13:20:22 GMT

John, Just to get it straight, if I am making two operations with MSMQ in the same transaction, I can expect the speed to drop to a crawl? That doesn't quite make sense. Just to give you an idea, I can do thousands of transactions pre sec using SQL Server + DTC. I don't understand why making two operations (on two queues on the same machine) will cause this major slow down. For that matter, I just tested it making two operations on the SAME queue, and I experience the same slow down. I would assume that making transactional operations is pretty important for MSMQ, and as far I as I can tell, behaviors like read from queue, send to queue in the same transaction are pretty common