http://ayende.comAyende @ RahienCopyright (C) Ayende Rahien 2004 - 2021 (c) 202660And what if you use a HybridDictionary ? With large collections, a Hashtable is more performant. The HybridDictionary uses internally a hashtable when the collection is large, and a ListDictionary when the collection is small. What are the results if you use a HybridDictionary ? http://ayende.com/3099/algorithms-joins-and-performance#comment7http://ayende.com/3099/algorithms-joins-and-performance#comment7Thu, 17 Jan 2008 11:31:14 GMTHave you tried Perfect Hashing? You know the keys in advance, and access them much more times then their count. You can get really fast lookup. http://ayende.com/3099/algorithms-joins-and-performance#comment6http://ayende.com/3099/algorithms-joins-and-performance#comment6Tue, 15 Jan 2008 09:35:09 GMTUsing DSA database was exactly what I refering to. In memory operations quickly become impossible if your size of your data grows. Then you would have to implement a lot of functionality most DB engines offer otb. But that ain't su much fun :) http://ayende.com/3099/algorithms-joins-and-performance#comment5http://ayende.com/3099/algorithms-joins-and-performance#comment5Tue, 15 Jan 2008 08:55:34 GMTI considered this, sure. See previous discussion on the blog. Right now I don't need it, but it is the logical next step, I think http://ayende.com/3099/algorithms-joins-and-performance#comment4http://ayende.com/3099/algorithms-joins-and-performance#comment4Mon, 14 Jan 2008 21:05:43 GMTHave you considered using a local DB as a temp store? At MP3.com we used unix join/sort/files/cut commands on text file for ETL because it beat in-memory operations. I recently talked to one of the warehouse guys from those days and he told me that he now uses SQL Lite as a local store for doing his transformations and found it to beat in memory operations. http://ayende.com/3099/algorithms-joins-and-performance#comment3http://ayende.com/3099/algorithms-joins-and-performance#comment3Mon, 14 Jan 2008 20:51:53 GMTBunter, In this case, I am doing a join of items from file into the results of a web service calls and then joining that to a DB call. The result goes to a second DB. The context is an ETL tool, not a DB http://ayende.com/3099/algorithms-joins-and-performance#comment2http://ayende.com/3099/algorithms-joins-and-performance#comment2Mon, 14 Jan 2008 18:52:25 GMTStrange that you are doing this in memory instead of having database handle it. http://ayende.com/3099/algorithms-joins-and-performance#comment1http://ayende.com/3099/algorithms-joins-and-performance#comment1Mon, 14 Jan 2008 18:26:19 GMT