Ayende @ Rahien

It's a girl

What am I missing? MSMQ Perf Issue

I am getting some strange perf numbers from MSMQ, and I can’t quite figure out what is going on here.

The scenario is simple, I have a process reading from queue 1 and writing to queue 2. But performance isn’t anywhere near where I think it should be.

In my test scenario, I have queue 1 filled with 10,000 messages, each about 1.5 Kb in size. My test code does a no op move between the queues. Both queues are transactional.

Here is the code:

private static void CopyData()
{
var q1 = new MessageQueue(@".\private$\test_queue1");
var q2 = new MessageQueue(@".\private$\test_queue2");
q2.Purge();
var sp = Stopwatch.StartNew();
while (true)
{
using (var msmqTx = new MessageQueueTransaction())
{
msmqTx.Begin();

Message message;
try
{
message = q1.Receive(TimeSpan.FromMilliseconds(0), msmqTx);
}
catch (MessageQueueException e)
{
Console.WriteLine(e);
break;
}

q2.Send(message, msmqTx);

msmqTx.Commit();
}
}
Console.WriteLine("{0:#,#}", sp.ElapsedMilliseconds);
}

Using this code, it takes 236.8 seconds to move 10,000 messages. If I use System.Transactions, instead of MSMQ’s internal transactions, I get comparable speeds.

Just to give you an idea, this is about 40 messages a second, this number is ridiculously low.

Changing the code so each operation is a separate transaction, like this:

private static void CopyData()
{
var q1 = new MessageQueue(@".\private$\test_queue1");
var q2 = new MessageQueue(@".\private$\test_queue2");
q2.Purge();
var sp = Stopwatch.StartNew();
while (true)
{

Message message;
try
{
message = q1.Receive(TimeSpan.FromMilliseconds(0), MessageQueueTransactionType.Single);
}
catch (MessageQueueException e)
{
Console.WriteLine(e);
break;
}

q2.Send(message, MessageQueueTransactionType.Single);
}
Console.WriteLine("{0:#,#}", sp.ElapsedMilliseconds);
}

Means that it takes 16.3 seconds, or about 600 messages per second, which is far closer to what I would expect.

This is on a quad core machine 8 GB RAM (4 GB free), so I don’t think that it is the machine that is causing the problem. I can see similar results on other machines as well.

Am I missing something? Is there something in my code that is wrong?

Comments

Tapio Kulmala
10/18/2009 11:37 AM by
Tapio Kulmala

Is there a similar speed difference if you have only one queue and you are only reading messages?

Tapio

Ayende Rahien
10/18/2009 11:40 AM by
Ayende Rahien

Tapio,

If I am just reading with I get 2331 messages per second using msmq transactions and about 3,000 messages per second using Single

AwkwardCoder
10/18/2009 11:54 AM by
AwkwardCoder

Correct me if I'm wrong but transactional queues pesist every message to disk before returning from Send, so therefore the amount of RAM has little affect and more likely disc IO could be an issue.

What's the performance like you just dump 10,000 1.5kb messages to disc directly?

Ayende Rahien
10/18/2009 12:05 PM by
Ayende Rahien

The code just here push 10,000 messages in 5.32 seconds, about 1,800 messages/sec

for (int i = 0; i < 10000; i++)

{

using (var msmqTx = new MessageQueueTransaction())

{


    msmqTx.Begin();


    q1.Send(new Message

    {

        BodyStream = new MemoryStream(bytes),

        Label = i.ToString()

    }, msmqTx);


    msmqTx.Commit();

}


if (i%10 == 0) Console.WriteLine(i);

}

Tapio Kulmala
10/18/2009 12:41 PM by
Tapio Kulmala

It would nice to profile the first example and see where the time is actually spent. Maybe the MessageQueueTransactionType.Single does not create a "real" transaction. You can't even rollback 'em or can't you? The first case creates a transaction that includes 2 resources (queues) while the second one creates 2 "transactions" each having only one queue. Maybe that has something to do with it. What happens if you create a transactionscope and use MessageQueueTransactionType.Auto?

Tapio

Ayende Rahien
10/18/2009 12:48 PM by
Ayende Rahien

Tapio,

Most of the time is spent on the Commit()

Same results for DTC

John Breakwell
10/18/2009 01:12 PM by
John Breakwell

Hi

"If I use System.Transactions, instead of MSMQ’s internal transactions, I get comparable speeds."

You are trying to perform two independent transactional operations within the same transaction so I expect it is using MSDTC to coordinate them (should be able to see the MSDTC transaction count to verify).

A single MSMQ transaction only applies to one operation, not both, so you'll see a much faster speed.

Cheers

John Breakwell (MSFT)

Ayende Rahien
10/18/2009 01:20 PM by
Ayende Rahien

John,

Just to get it straight, if I am making two operations with MSMQ in the same transaction, I can expect the speed to drop to a crawl?

That doesn't quite make sense.

Just to give you an idea, I can do thousands of transactions pre sec using SQL Server + DTC.

I don't understand why making two operations (on two queues on the same machine) will cause this major slow down.

For that matter, I just tested it making two operations on the SAME queue, and I experience the same slow down.

I would assume that making transactional operations is pretty important for MSMQ, and as far I as I can tell, behaviors like read from queue, send to queue in the same transaction are pretty common

Chris Patterson
10/18/2009 03:05 PM by
Chris Patterson

I noticed something very similar Ayende.

With Windows Server 2003 using .Single I would easily get 2k messages a second with load tests. On Server 2008/w7 I get on the order of maybe 350 pt second. Running it under dotTrace, all of the time is spent in MSMQ (my total application touch time is less than 500ms compared to like 22 seconds in MSMQ).

I figure it had to be something to do with the transaction support on 2k8/vista/7 compared to 2003 server. But you're right, it's dog slow on the newer OS.

Ayende Rahien
10/18/2009 03:07 PM by
Ayende Rahien

Chris,

We are seeing something similar on Win2003 Server as well, though

Frank Quednau
10/18/2009 04:22 PM by
Frank Quednau

Just for comparison, on my machine, 32-bit Dual core, 3GB RAM it takes 162 seconds. Not superfast, but about a minute faster than your measurement.

This is on Vista...haven't used MSMQ for a while so I had to do a fresh MSMQ feature install. Installed without any addons like triggers HTTP support, etc. - just the core.

Reducing the commit load by always reading and then writing 10 messages in a single transaction literally slahed the time to 1/10th, showing that the commit indeed is by far the most costly operation in the code.

Rafal
10/18/2009 06:18 PM by
Rafal

Wow, quite shocking. I'm using custom message queuing implementation based on a table in SQL server database and I'm getting about 150 messages per second (while enclosing message handler in a TransactionScope). And I was wondering if moving to msmq will allow me to handle thousands messages per second - looks like i wouldn't even be able to have current performance.

@Frank - the problem is that you rarely can send or receive messages in batches, usually each message is an individual transaction.

@Ayende - I suspect this also affects RSB performance as RSB is using distributed transactions?

Ayende Rahien
10/18/2009 08:01 PM by
Ayende Rahien

Rafal,

Yes, this affects RSB, because RSB make use of MSMQ.

I still need to test how RQ behaves here.

J Healy
10/18/2009 09:37 PM by
J Healy

I'm guessing RhinoMQ and RhinoTX will be in extended beta sometime tomorrow afternoon...

Ayende Rahien
10/18/2009 09:40 PM by
Ayende Rahien

J Healy,

You are aware that there IS Rhino Queues project, right?

J Healy
10/18/2009 10:06 PM by
J Healy

Ah, that's right - I've clearly been distracted over the summer and need to catch up with things a bit. As I [vaguely] recall you were going to get Rhino SB running on top of your Queues, did that happen?

Ayende Rahien
10/18/2009 10:19 PM by
Ayende Rahien

J Healy,

Yes, RSB works with RQ

Steve Py
10/18/2009 11:20 PM by
Steve Py

I think the trouble is that your transaction is guarding an operation against a queue that is also receiving messages. The premise is that if something fails, the record from the receive transaction will be "put back", and this might be causing your performance headache.

From your example I don't believe a transaction is required. If the order of the received messages is important then use Peek, attempt to process, and if successful (or determined to be poisoned) pull it off the queue (I.e. ReceiveByID). If the order of messages isn't important read it, attempt to process, then if it fails, decide whether to pop it back on to try again, or discard.

Transactions carry a pretty steep performance hit in MSMQ and are really geared towards coordinating multiple queues.

Rafal
10/19/2009 07:55 AM by
Rafal

@Steve - you probably are right, there's no need to enclose everything in a distributed transaction if the only thing you have to do is to put message back into the queue. You could use 'local' msmq transaction for that purpose and a separate, distributed transaction for the message handler.

Ayende Rahien
10/19/2009 08:05 AM by
Ayende Rahien

Steve,

What happens if my process crashes? A message is lost

And Peek will let two concurrent consumers read the same message (BAD!)

I am also trying to coordinate between two queues, so I want to make use of this.

Ayende Rahien
10/19/2009 08:05 AM by
Ayende Rahien

Rafal,

In the test code, I AM using local MSMQ transaction

Rafal
10/19/2009 10:10 AM by
Rafal

Ayende, thanks for pointing this out, it's worse than I thought. But there's a post on Udi Dahan's blog, regarding NServiceBus performance ( www.udidahan.com/.../nservicebus-performance/) and he says:

"OK, so using the default nServiceBus distribution using MSMQ, on servers where the queue files themselves were on separate SCSI RAID disks, we were pumping around 1000 durable, transactionally processed messages per second, per server."

Ayende Rahien
10/19/2009 11:11 AM by
Ayende Rahien

Rafal,

Yes, that is why I am shocked by this.

Kerry Jenkins
10/19/2009 06:45 PM by
Kerry Jenkins

Would it meet your requirements to create one transaction for all of the moves? I tried this by moving the "while (true)" statement down to after the msmqTx.Begin and moved the msmqTx.Commit after the ending "while" brace. This dramatically sped up the processing. I tested what an abort would do where the commit would normally happen after 10000 records were moved. It was very fast as well.

Ayende Rahien
10/19/2009 06:52 PM by
Ayende Rahien

Kerry,

No, this is just test code.

I am trying to see why my app (which does something similar) acts this way

Jesse Ezell
10/19/2009 10:22 PM by
Jesse Ezell

Yes. MSMQ transactions are EXTREMELY slow. I have seen similar behavior. To get decent numbers with transactions, you must use something like transaction batching. Of course, if you just switch to WCF and net.msmq binding, this is provided out of the box.

Steve Py
10/19/2009 10:23 PM by
Steve Py

I've never had to try and use transactions in this manner. You're likely going to get nailed trying to commit a change to a Queue that is actively receiving messages. My suggestion around Peek was if the order was significant which would imply 1 listener.

As for losing messages, that is an exceptional circumstance. Could it be handled by the poster of the message if it's expecting a response? MSMQ exposes quite a bit of functionality such as PeekById and prioritization to help detect and re-send dropped messages. You may want to reconsider whether the cost of attempting to guarantee delivery is worth it, or just handle the exceptional case. (ala using UDP vs TCP)

I don't know if this will work, but have you tried moving the creation of the transaction instance itself outside of the loop? (Still do the Begin and Commit inside the loop for each message, but avoid recreating a new transaction instance each iteration.) Not sure if transactions can be re-used like this or not...

Ayende Rahien
10/19/2009 11:47 PM by
Ayende Rahien

Steve,

Consider the simple case of:

tx.begin

ordermsg = readorderfromqueue

if processmsg ordermsg:

send_msg_to_warehouse ship_order

tx.commit

I have a lot of logic that does this. If I crash, I want things to recover properly.

If I need to manage stuff manually, I might as well drop transactions all together.

Ayende Rahien
10/19/2009 11:48 PM by
Ayende Rahien

Jesse,

Can you give me a link to this, I am not familiar with the term

Gerke Geurts
10/19/2009 11:59 PM by
Gerke Geurts

I remember similar perf issues a long time ago with MSMQ transactions that were caused by slow write caching of the RAID controller. Replacement of the RAID controller improved a lot. Maybe MSMQ transactions are rather sensitive to disk write latency.

Jesse Ezell
10/20/2009 12:52 AM by
Jesse Ezell

Here's how to do it with WCF.

msdn.microsoft.com/en-us/library/aa395219.aspx

You can do the same type of thing manually with MSMQ. The big slowdown appears to be due to the transactions themselves, so if you can send multiple messages in a single transaction, you will get significantly better performance.

Steve Py
10/20/2009 01:46 AM by
Steve Py

Yeah, in that scenario I would advise having the sender manage contingency for it's message not being received and processed rather than transactions. I don't seen anything with what you've done that might suggest why the transaction is more expensive than it should be. (Other than maybe disk latency as Gerke mentioned.)

Services where I use MSMQ always have their own ACK private queue available for acknowledgements / status updates of their respective messages that they pass the address along as part of the request message. If they don't get an ACK to their satisfaction, they ping the queue for their request (in case it's still pending) and re-send if they don't find it. It's simple to wrap in, and if the scenario only manifests itself <2% of the time, it's a lot better option than accepting a 15-50% (or larger) performance hit overall.

Of course if you're looking for "fire & forget" where the poster can't be assumed to be responsible for checking on its own requests, then either accept the performance hit of the transaciton, or consider using a database instead.

Where I would use transactions is if I needed to ensure a message makes it onto multiple queues successfully, or situations where messaging is used to transmit sequential messages or messages on demand between 2 or more points. Chain is satisfied, commit.

Ah well, good luck in any case. ;)

Ayende Rahien
10/20/2009 10:02 AM by
Ayende Rahien

Steve,

Yes, I'll probably have to go this way, because of the horrible perf of tx in MSMQ.

Very disappointed, thanks for the ideas

David
10/20/2009 04:54 PM by
David

I don't know if it's true with MSMQ 4.0, but with earlier versions, having journaling enabled significantly increases overhead when receiveing messages from a queue.

Alois Kraus
10/21/2009 08:09 PM by
Alois Kraus

One very obvious thing:

With what serializer do you create your msmq messages? The default (xml) is by far the slowest one. You should set it explicitely to get most out of it.

Yours,

Alois Kraus

Ayende Rahien
10/21/2009 08:29 PM by
Ayende Rahien

Alois,

This is using a stream to do stuff, not a serializer.

Lothan
10/22/2009 05:02 AM by
Lothan

It's definitely the single-item transaction, Ayende. I copied your initial code and then modified it to perform two runs. The first run without transactions:

SendData: 269 milliseconds

CopyData: 554 milliseconds

I then wrapped the loops in a transaction, so I'm committing all 10,000 messages in a single transaction:

SendData: 481 milliseconds

CopyData: 1,944 milliseconds

Your original code that commits each item in a separate transaction:

SendData: 16,107 milliseconds

CopyData: 194,201 milliseconds

So, yeah, the transaction cost here is astronomical and seems to have some sort of exponential function.

Comments have been closed on this topic.