http://ayende.comAyende @ RahienCopyright (C) Ayende Rahien 2004 - 2021 (c) 202660See my latest post about Rhino ETL for the answer. http://ayende.com/3083/pipes-and-filters-the-multi-threaded-version#comment8http://ayende.com/3083/pipes-and-filters-the-multi-threaded-version#comment8Wed, 16 Jan 2008 07:31:22 GMTHi Ayende, I like your pipe and filter implementation, its super concise and effective. But I'm curious how you intend to implement forks and joins into your pipe and filter pattern. I'd like to see you try this out :) http://ayende.com/3083/pipes-and-filters-the-multi-threaded-version#comment7http://ayende.com/3083/pipes-and-filters-the-multi-threaded-version#comment7Tue, 08 Jan 2008 15:04:49 GMTI've said it before but I'll say it again: you really need to go catch up on the ParallelFX libraries. The CTP is available and there are great videos on Channel9. You'll find parallel for loops, parallel foreach, parallel LINQ (over objects at the least, not sure about the others), tasks, futures, all with configurable parallelization, work-stealing, ordering, etc. It is well worth checking out, if only to help distinguish your goals and approach from theirs. It'd help us readers for that reason, too. ;) http://ayende.com/3083/pipes-and-filters-the-multi-threaded-version#comment6http://ayende.com/3083/pipes-and-filters-the-multi-threaded-version#comment6Sun, 06 Jan 2008 18:46:36 GMTJon, If I never have to write multi threaded code again, I'll be very happy. See my previous response to pipelines & linq in the previous thread. I don't think that you can scale the syntax to be appropriate. I would love to see myself proved wrong, however. http://ayende.com/3083/pipes-and-filters-the-multi-threaded-version#comment5http://ayende.com/3083/pipes-and-filters-the-multi-threaded-version#comment5Sun, 06 Jan 2008 15:05:30 GMTHaving rediscovered LINQ as pipes and filters, I think you've now rediscovered Parallel LINQ :) Seriously, it would be worth looking into LINQ further, and the Parallel Extensions. There's a CTP available at http://www.microsoft.com/downloads/details.aspx?FamilyId=E848DC1D-5BE3-4941-8705-024BC7F180BA&displaylang=en Without in any way wishing to cast aspersions on your multithreading abilities, the guys behind Parallel LINQ have spent a long time on it and are seriously smart on concurrency. I'm sure they'd value your input, too. Jon http://ayende.com/3083/pipes-and-filters-the-multi-threaded-version#comment4http://ayende.com/3083/pipes-and-filters-the-multi-threaded-version#comment4Sun, 06 Jan 2008 15:01:49 GMTIn my book Practical .NET2 and C#2 I explain how to code pipeline with iterators of C#2 (keyword yield and yield break). The result is super concise syntax. For example: http://www.practicaldot.net/Chapter_14/Listing_14_47.htm And also how to compute prime numbers with pipes with not even 20 lines of code. http://www.practicaldot.net/Chapter_14/Listing_14_50.htm The whole chapter is available as an article here: http://65.214.43.45/tt/articles/showarticle.tss?id=IteratorsWithC2 http://ayende.com/3083/pipes-and-filters-the-multi-threaded-version#comment3http://ayende.com/3083/pipes-and-filters-the-multi-threaded-version#comment3Sun, 06 Jan 2008 10:05:58 GMTForgot to put this link: http://labs.google.com/papers/mapreduce.html For the map reduce... http://ayende.com/3083/pipes-and-filters-the-multi-threaded-version#comment2http://ayende.com/3083/pipes-and-filters-the-multi-threaded-version#comment2Sun, 06 Jan 2008 06:43:48 GMTTake a look at Google's map-reduce. It really resembles what u showed here, especially the multi threaded version, except that they use it for massive computations on grids of computers. I think that with a little remoting or WCF your pipeline implementation can do something similar. http://ayende.com/3083/pipes-and-filters-the-multi-threaded-version#comment1http://ayende.com/3083/pipes-and-filters-the-multi-threaded-version#comment1Sun, 06 Jan 2008 06:42:38 GMT