http://ayende.comAyende @ RahienCopyright (C) Ayende Rahien 2004 - 2021 (c) 202660@Ayende, "This is a failure scenario, where the first consumer has failed" >> That is one possible scenario, other could be where the first consumer Really processed the message, but took longer than timeout. As we are not using distributed transactions, there could be actual work committed to the db. In this case, second consumer may actually fail. http://ayende.com/4678/queuing-systems#comment19http://ayende.com/4678/queuing-systems#comment19Wed, 03 Nov 2010 20:01:12 GMTManinder, a) this is typical for remote queues. b) this is a failure scenario, where the first consumer has failed http://ayende.com/4678/queuing-systems#comment18http://ayende.com/4678/queuing-systems#comment18Wed, 03 Nov 2010 19:34:29 GMT@Ayende, I said "And if multiple processes are connected to same queue, it is also possible for multiple consumers to consume the same message under certain conditions." The key is "under certain conditions". So if there are multiple processes on the host machine, picking up messages from the local queue, then it is possible under the model of "Client Acknowledgement" as opposed to "Auto Acknowledgement" that the consuming process could not respond with ACK, within timeout period and hence, queuing system put the message back on the queue , which gets picked by the second consumer. Hence, multiple consumer , under certain conditions can process the same message even on a local queue scenario. http://ayende.com/4678/queuing-systems#comment17http://ayende.com/4678/queuing-systems#comment17Wed, 03 Nov 2010 19:30:35 GMTManinder, Even in the same process, things can fail. For example HD space might have run out. But in general, when speaking about such things, it is common to think of a single machine as a single unit. In the same sense that you are composed of a complex system of organs working together, but you consider yourself a single entity. And I am not aware of a single queuing impl. that would let multiple consumers consume the SAME message. Note that there are things like topics, streams and exchanges, but that isn't what I am talking about here. http://ayende.com/4678/queuing-systems#comment16http://ayende.com/4678/queuing-systems#comment16Wed, 03 Nov 2010 17:58:01 GMT@Ayende, I am unclear on your consequences around local queue. >> Local queue is always going to be available – in other words: queue.Send() cannot fail. My understanding is that a Local Queue is Local to the machine and not the process. Hence, based on the client implementation of Local Queue and the underlying operating system, you could use tcp. If for example the host system runs out of file descriptors, the send() and receive for local can fail also. >> Only one consumer can process a message. You can have multiple processes connected to a local queue on the same machine. And if multiple processes are connected to same queue, it is also possible for multiple consumers to consume the same message under certain conditions. >>There is a lot of state held locally. State is held in the queuing system and not the processing system. Irrespective of remote or local, the state will be held. Also based on queue system implementation, you can design for failovers. http://ayende.com/4678/queuing-systems#comment15http://ayende.com/4678/queuing-systems#comment15Wed, 03 Nov 2010 17:53:14 GMT@Ayende "it is important to run the distributor on a cluster configuring its queues as clustered resources" We're looking at the definition of a local queue differently. You listed 5 things in your article that distinguish a local queue from a remote queue - I thought that a queue configured as a clustered resource would meet those requirements. http://ayende.com/4678/queuing-systems#comment14http://ayende.com/4678/queuing-systems#comment14Mon, 01 Nov 2010 14:09:05 GMTAyende, u should really have a look into ZooKeeper( [http://hadoop.apache.org/zookeeper/](http://hadoop.apache.org/zookeeper/)), It is not just queue, but have more features that the other lacking.., and probably could give u some idea http://ayende.com/4678/queuing-systems#comment13http://ayende.com/4678/queuing-systems#comment13Mon, 01 Nov 2010 03:28:13 GMTBrian, Yes, it does. The distributer simply split the process into several steps, each of them assumes a local queue. http://ayende.com/4678/queuing-systems#comment12http://ayende.com/4678/queuing-systems#comment12Sun, 31 Oct 2010 16:27:41 GMTNServiceBus doesn't always assume the queue you have is a local one: [http://www.nservicebus.com/Distributor.aspx](http://www.nservicebus.com/Distributor.aspx)http://ayende.com/4678/queuing-systems#comment11http://ayende.com/4678/queuing-systems#comment11Sun, 31 Oct 2010 14:49:21 GMT@Jeff AMQP (specifically RabbitMQ) have great .NET support these days. ZeroMQ also looks like it has a lot of potential buti have yet to try it. http://ayende.com/4678/queuing-systems#comment10http://ayende.com/4678/queuing-systems#comment10Sun, 31 Oct 2010 01:19:16 GMTIt's a bummer .NET doesn't have a decent native messaging solution like java does (JMS implementations abound). MSMQ is a sorrow excuse for an enterprise messaging solution IMO. Apache.NMS ( [http://activemq.apache.org/nms.html](http://activemq.apache.org/nms.html)) provides a decent client for .NET interaction with ActiveMQ. I have been using it quite a bit lately and rocks. It supports all the major features of JMS including transactions, durable consumers, topics, queues, different ack modes and so on and so forth. http://ayende.com/4678/queuing-systems#comment9http://ayende.com/4678/queuing-systems#comment9Sat, 30 Oct 2010 23:08:26 GMTAyende, you mentioned Amazon SQS. Can you deploy local queues in a cloud? Is it only a cost of deploying a queuing system or maybe mismatch in paradigms? http://ayende.com/4678/queuing-systems#comment8http://ayende.com/4678/queuing-systems#comment8Sat, 30 Oct 2010 19:26:42 GMTHave you taken a look at Sql Service Broker? You get transactions (msg + database activity), and free remoting with Sql Express. http://ayende.com/4678/queuing-systems#comment7http://ayende.com/4678/queuing-systems#comment7Sat, 30 Oct 2010 16:38:43 GMT@Dennis - the Quartz.NET project ( [http://quartznet.sourceforge.net/](http://quartznet.sourceforge.net/)) is a scheduler for .Net, if that suits your needs. http://ayende.com/4678/queuing-systems#comment6http://ayende.com/4678/queuing-systems#comment6Sat, 30 Oct 2010 16:28:13 GMTWhat about scheduling system for csharp? Almost any major website can use that for doing some work at some specific time. http://ayende.com/4678/queuing-systems#comment5http://ayende.com/4678/queuing-systems#comment5Sat, 30 Oct 2010 14:23:56 GMT@Bill You need to either make your messages indempotent or have some sort of tracking system (such as a database) to keep an eye out for duplicates. Any other approach I've tried has not worked out well. http://ayende.com/4678/queuing-systems#comment4http://ayende.com/4678/queuing-systems#comment4Sat, 30 Oct 2010 13:18:13 GMTyes, thats one way of doing it but as you mention is not a real transaction i.e not 100% reliable. Its an interesting problem since the tradeoffs are performance for transactional safety. There are some java solutions which look good but I am not sure about their performance compared to msdtc. I have for a while thought about looking in to java queuing and transactions since the options on the Microsoft side are limited. http://ayende.com/4678/queuing-systems#comment3http://ayende.com/4678/queuing-systems#comment3Sat, 30 Oct 2010 12:12:35 GMTBill, Generally, remote queuing systems can't really take part of a real transaction. Mostly because it costs too much. What you can do is set the timeout for a very long time, and ack on the transaction commit. http://ayende.com/4678/queuing-systems#comment2http://ayende.com/4678/queuing-systems#comment2Sat, 30 Oct 2010 11:08:54 GMTInteresting, I was just doing some work with MSDTC and MSMQ remote queues. With the poison message scenario you describe (as an alternative to MSDTC) how would rollback be handled in the case of a larger transaction? What if there was a db operation and a queue operation that needed to be transactional but the db operation failed on commit? It seems like the poison message method only works if you are only using a message queue and nothing else is taking part in the transaction? Would be interested to know if those scenarios are handled as they are quite common in large scale non trivial systems which is where MSDTC is of use. http://ayende.com/4678/queuing-systems#comment1http://ayende.com/4678/queuing-systems#comment1Sat, 30 Oct 2010 10:42:44 GMT