http://ayende.comAyende @ RahienCopyright (C) Ayende Rahien 2004 - 2021 (c) 202660Includes and live projections are useful for retrieving related data, but they don't help if you need to query or sort on the related data. Denormalised references are an option, but they're not always appropriate, e.g. if the related data has a high rate of change. That's why I'm really looking forward to this feature. It perfectly fills the gap left by denormalised references, includes and live projections. Well done to Frank for coming up with the idea!http://ayende.com/89089/ravendb-multi-maps-reduce-indexes#comment15http://ayende.com/89089/ravendb-multi-maps-reduce-indexes#comment15Wed, 14 Sep 2011 15:00:14 GMTChanan, You can do that, certainly. It is a little different, but it is entirely feasiblehttp://ayende.com/89089/ravendb-multi-maps-reduce-indexes#comment14http://ayende.com/89089/ravendb-multi-maps-reduce-indexes#comment14Tue, 13 Sep 2011 08:13:57 GMTTobi, You don't relaly need to denormalize often in RavenDBhttp://ayende.com/89089/ravendb-multi-maps-reduce-indexes#comment13http://ayende.com/89089/ravendb-multi-maps-reduce-indexes#comment13Tue, 13 Sep 2011 08:13:15 GMTTobi, No, the actual problem is how do you find what you join ON. Remember, we don't have a column for that join condition, this can be anywhere you want it to. And just getting the data to do the join would be incredibly expensive.http://ayende.com/89089/ravendb-multi-maps-reduce-indexes#comment12http://ayende.com/89089/ravendb-multi-maps-reduce-indexes#comment12Tue, 13 Sep 2011 08:11:50 GMTfschwiet, the ability to use multiple sources is a union-all. It is clearly useful especially for search. It has nothing to do with joins inherently, it is just that you can assemble/hack a join from it by doing a reduce stage afterwards. If you want to know how joins would look like, google DryadLINQ. It is a data-warehouse query solution, not OLTP. Yet it is very applicable and instructive to this discussion.http://ayende.com/89089/ravendb-multi-maps-reduce-indexes#comment11http://ayende.com/89089/ravendb-multi-maps-reduce-indexes#comment11Tue, 13 Sep 2011 07:51:08 GMTChanan, I gave it a try. I only had a hour or so to try it out, it seems possible but I couldn't quite get it to work: https://github.com/fschwiet/ravendb/commit/597c8630360e6a28331e9f738ac5e753807f4a46 One observation, to group by a field it needs to be included in all types that are mapped, (for Ayende's test, I had to add UserName field to Post in order to group by first letter of username) Tobi, you just have to write a distributed map/reduce function in linq. Thats much better than say, Erlang. :) What would the joins look like? Trying multimaps, there is room to make it more DRY when coalescing, but being able to merge different types like this does open some possibilities for composite views, very typical on the web. http://ayende.com/89089/ravendb-multi-maps-reduce-indexes#comment10http://ayende.com/89089/ravendb-multi-maps-reduce-indexes#comment10Mon, 12 Sep 2011 19:59:23 GMTSomewhat related question... Do map/reduce always return a key and a count? Or in other words, for example, I want to return the first letter of the last name of the authors and group of authors: B: Braunstein, Chanan Bova, Ben R: Rahien, Ayendehttp://ayende.com/89089/ravendb-multi-maps-reduce-indexes#comment8http://ayende.com/89089/ravendb-multi-maps-reduce-indexes#comment8Mon, 12 Sep 2011 14:48:30 GMTNeedless to say that simple standard joins would remove the need to denormalize in RavenDB and be a great convenience. I expect the NoSql fad to end when somebody finally imlements a relational database with relaxed guarantees. RavenDB could be that database right now. Basically, only convenient joins and a reasonably query planner are missing.http://ayende.com/89089/ravendb-multi-maps-reduce-indexes#comment7http://ayende.com/89089/ravendb-multi-maps-reduce-indexes#comment7Mon, 12 Sep 2011 14:28:28 GMTAyende, the implementation problem will probably be a) the change tracking and b) the need for a more efficient join than loop-joins for view initialization. Difficulty is in this order. Implementing an external merge sort + merge-join is easy, I have done it to process "big data" (more than RAM would hold). The change tracking will be difficult to work out but easy to test. A reference implementation exists in SQL Server and can be analyzed by looking at the execution plans. They are detailed enough to reverse-engineer a solution. http://ayende.com/89089/ravendb-multi-maps-reduce-indexes#comment6http://ayende.com/89089/ravendb-multi-maps-reduce-indexes#comment6Mon, 12 Sep 2011 14:05:15 GMTTobi, Try to implement that, and tell me where it leads you. We have spent a LOT of time on that, and this is by far the simplest solutionhttp://ayende.com/89089/ravendb-multi-maps-reduce-indexes#comment5http://ayende.com/89089/ravendb-multi-maps-reduce-indexes#comment5Mon, 12 Sep 2011 13:32:37 GMTJohn, thanks, fixedhttp://ayende.com/89089/ravendb-multi-maps-reduce-indexes#comment4http://ayende.com/89089/ravendb-multi-maps-reduce-indexes#comment4Mon, 12 Sep 2011 13:30:51 GMT@John: I think Count = 1 is just a way to initialize the property Count of the anonymous type to an int32.http://ayende.com/89089/ravendb-multi-maps-reduce-indexes#comment3http://ayende.com/89089/ravendb-multi-maps-reduce-indexes#comment3Mon, 12 Sep 2011 13:21:28 GMTThis is a pretty specialized and nasty way to do materialized views over join queries. I don't know why you resist joins so much. You just _implemented_ materialized joins, but in a totally involved way because it had to be forced into the map/reduce paradigm. There is nothing in joins that is inherently unscalable or unperformant. Look at how SQL Server does incremental and efficient maintainance of indexed views with joins in them. Querying them has zero runtime overhead yet you get all the denormalization benefits and efficient updates. I totally don't understand why all map/reduce implementations in the entire industry just cannot admit that joins are useful and should be supported. DryadLINQ is the welcome exception. http://ayende.com/89089/ravendb-multi-maps-reduce-indexes#comment2http://ayende.com/89089/ravendb-multi-maps-reduce-indexes#comment2Mon, 12 Sep 2011 11:07:10 GMTI think the line in your name/posts example should read: select new { UserId = user.Id, Author = user.Name, Count = 0 } Otherwise your count will always be one too many.http://ayende.com/89089/ravendb-multi-maps-reduce-indexes#comment1http://ayende.com/89089/ravendb-multi-maps-reduce-indexes#comment1Mon, 12 Sep 2011 09:25:40 GMT