Ayende @ Rahien

h commented on Transactional queuing system perf test

Fri, 06 Nov 2009 01:01:33 GMT

I can recommend trying out RabbitMQ as well and having a look at erlang ;).

Frank commented on Transactional queuing system perf test

Fri, 30 Oct 2009 18:06:35 GMT

Hmm, did some further testing, resulting in the following observations: - Sending one message per distributed transaction is fast. - Receiving one message per distributed transaction is fast. - Receiving one message and sending it to another queue per distributed transaction is very slow (over 10 times slower than the other scenarios). - All scenarios are actually using MSDTC as can be seen from the number of distributed transaction that succeeded increase. That receiving and sending is a creepy one. :(

Steve Py commented on Transactional queuing system perf test

Fri, 30 Oct 2009 10:39:11 GMT

@Michal No, this post is probably a bit confusing without referring back to his previous gripe about MSMQ transaction performance. Writing within a transaction is plenty fast. His last MSMQ example where he hasn't read or written was still bloody slow for some reason. I believe MSMQ is just behaving a bit shirty because no operation has been taken on a Queue when the transaction has been committed. His ultimate goal is determining why reading *and* writing within the same transaction is so slow. P.s. The above example I posted had a 2nd method that populated the first private queue with 1000 messages as per Ayende's first example. I'm at home now for the weekend so I have the time and resources to play around with queues again. :)

MichaelGG commented on Transactional queuing system perf test

Fri, 30 Oct 2009 10:34:07 GMT

After using MSMQ a few times and then discovering SQL Server Service Broker, I cannot fathom ever having to use MSMQ again. SSSB is much easier to setup and monitor. And since it is accessible just as extensions to T-SQL, it's accessible. And if your queue processing code ends up working against the local database, you won't need a distributed transaction.

Steve Py commented on Transactional queuing system perf test

Fri, 30 Oct 2009 10:29:39 GMT

Ok, I found something that might help explain the behaviour: [blogs.sun.com/.../java_caps_tip_reading_msmq](http://blogs.sun.com/toxophily/entry/java_caps_tip_reading_msmq) The issue is reading the message within the transaction. For example if you write your method to transfer 1000 messages from one transactional queue to the other, and utilize a Peek within the transaction and a Receive *after* the commit, your performance shoots up to respectable speeds... private static void TransferTx() { var sp = Stopwatch.StartNew(); int msgs = 0; using ( var msmqFrom = new MessageQueue( @".\Private$\test" ) ) { using ( var msmqTo = new MessageQueue( @".\Private$\test2" ) ) { using ( var msmqTx = new MessageQueueTransaction() ) { for ( int i = 0; i < 1000; i++ ) { msmqTx.Begin(); try { var message = msmqFrom.Peek( TimeSpan.Zero ); msmqTo.Send( new System.Messaging.Message( "Test" + i ), msmqTx ); msgs = msgs + 1; if ( msgs % 100 == 0 ) Console.WriteLine( msgs ); msmqTx.Commit(); msmqFrom.Receive(); } catch { msmqTx.Abort(); } } } msmqTo.Purge(); } } Console.WriteLine( "Transfer Queues = {0:#,#}", sp.ElapsedMilliseconds/1000f ); } You can see the trouble with the receive from 1 queue and write to another. If you "Receive" with a transaction type of "none", effectively performing it outside of the transaction the performance shoots up. Based on that article your choices are a Peek which negates the transactional effect while preserving the behaviour. The message is received outside once the transaction is committed. The downside is of course this is only suitable for 1 listener on the receive queue. Of course if your listener is effectively working as a dispatcher moving messages between queues, this should be able to be modelled into a single listener role. The only other real alternative to minimize the impact of reading in a transaction is processing groups of messages within a single transaction.

Michal commented on Transactional queuing system perf test

Fri, 30 Oct 2009 07:52:10 GMT

Not really related to forwarding perf, but in the "10,000 messages separate transactions" scenario you seem to compare 1600 msg/sec in MSMQ and 45(60) msg/sec in RQ and write that RQ's "slightly" worse. Am I missing something or is there a typo?

Steve Py commented on Transactional queuing system perf test

Fri, 30 Oct 2009 02:03:59 GMT

Ah, pity. :)

Lothan commented on Transactional queuing system perf test

Fri, 30 Oct 2009 00:55:29 GMT

I just tried that Steve and from what I am seeing it does not make much difference whether the transaction is instantiated and disposed outside the loop or inside the loop.

Frank commented on Transactional queuing system perf test

Thu, 29 Oct 2009 23:38:05 GMT

What am I overlooking here? When I do some benchmarks on sending messages with MSMQ using distributed transactions, I do over 500 messages/second. This is the code. class Program { static void Main(string[] args) { const int messageSize = 1024; var messageData = new byte[messageSize]; for (int i = 0; i < messageSize; ++i) { messageData[i] = (byte)(i & 0xFF); } var sp = Stopwatch.StartNew(); for (int i = 0; i < 10000; i++) { using (var tx = new TransactionScope()) { using (var queue = new MessageQueue(@".\private$\Benchmark", QueueAccessMode.Send)) { var message = new Message(); message.BodyStream = new MemoryStream(messageData); queue.Send(message, MessageQueueTransactionType.Automatic); } if (i % 100 == 0) Console.WriteLine(i); tx.Complete(); } } sp.Stop(); Console.WriteLine("{0:#,#}", sp.ElapsedMilliseconds); } }

Steve Py commented on Transactional queuing system perf test

Thu, 29 Oct 2009 21:41:09 GMT

@ Ayende, I'd hinted at this option in your earlier post; Sorry, normally I'd try this myself but my work environment here doesn't have MSMQ set up; Have you tried this variation? (1 re-used transaction instance, individual begin/commits) private static void JustTx() { var sp = Stopwatch.StartNew(); int msgs = 0; using (var msmqTx = new MessageQueueTransaction()) { for (int i = 0; i < 1000; i++) { msmqTx.Begin(); msgs = msgs + 1; if (msgs % 10 == 0) Console.WriteLine(msgs); msmqTx.Commit(); } } Console.WriteLine("{0:#,#}", sp.ElapsedMilliseconds); } From what I've read in the doco it should work and sounds like it'd solve your Dispose issue.

Bryan commented on Transactional queuing system perf test

Thu, 29 Oct 2009 12:57:02 GMT

We don't use transactions when sending our messages. I know, crazy. We delay them until after database transaction commit, and then send them. Sure, once in a blue moon a process crashes or the messaging system goes down and we lose messages. This happens so rarely for us, that it has yet to cause a problem, but it's obviously still a risk. We then have cleaner processes that either rebroadcast messages and/or simply clean up any bad data. We usually need these processes regardless of our messaging subsystem anyway. So far, this has worked out pretty well for us and doesn't suffer from the performance problems of using a distributed transaction.

Rafal commented on Transactional queuing system perf test

Thu, 29 Oct 2009 11:36:08 GMT

I do hope so. I find it extremely strange that an empty MSMQ transaction takes longer to execute than a transaction actually doing something. Maybe you should include a call to 'dummyQueue.Send("whatever")' in each transaction?

Ayende Rahien commented on Transactional queuing system perf test

Thu, 29 Oct 2009 11:26:16 GMT

Rafal, Yes & no. It is a problem, but not an unsolvable one.

Rafal commented on Transactional queuing system perf test

Thu, 29 Oct 2009 11:14:01 GMT

This makes RSB load balancing feature quite useless, as the load balancer is unable to distribute messages faster than message handlers process them. Right?

annon commented on Transactional queuing system perf test

Thu, 29 Oct 2009 10:32:52 GMT

Did you try this with any AMQP - for example RabbitMQ( [http://www.rabbitmq.com/](http://www.rabbitmq.com/)) ? We switched from MSMQ to RabbitMQ and couldn't be more happier.