http://ayende.comAyende @ RahienCopyright (C) Ayende Rahien 2004 - 2021 (c) 202660Hm I did not know about the With method. I believe your query does not work because result.Album can be null so grouping by it makes no sense. It can be fixed however and that is all that counts. In general you can get a joined index with the following steps: var map = orders.Select(x => new { o = x, c = null }).Concat(customers.Select(x => new { o = null, c = x })); var reduce = map.GroupBy(x => x.o, (order, group) => new { order, group.FirstOrDefault() }; Probably you can construct a helper method that constructs such a query by using the expression api. That would restore the convenience factor. http://ayende.com/4650/nosql-without-web-scale#comment36http://ayende.com/4650/nosql-without-web-scale#comment36Thu, 07 Oct 2010 10:57:26 GMTTobi, Yes, it could. It would be somewhat more awkward, but... // map from orderOrLine in docs.With("Orders", "OrderLines") let order = orderOrLine.Is("Orders") ? orderOrLine : null let line = orderOrLine.Is("OrderLines") ? orderOrLine : null select new { Album = order == null ? line.Album : null, Quantity = order == null ? line.Quantity : 0, Customer = order == null ? null : order.Customer } // reduce from result in results group result by result.Album into g select new { Album = g.Key, Quantity = g.Sum(x=>x.Quantity), Customer = g.First(x=>x.Customer != null).Customer } http://ayende.com/4650/nosql-without-web-scale#comment35http://ayende.com/4650/nosql-without-web-scale#comment35Wed, 06 Oct 2010 21:17:19 GMT"you cannot combine data from different tables automatically" I have seen your example ( [ayende.com/.../...ting-the-homecontroller-the.aspx](http://ayende.com/Blog/archive/2010/05/21/porting-mvc-music-store-to-raven-porting-the-homecontroller-the.aspx)) but only one table is involved in the map reduce index. If the order lines were a separate entity the index could not be constructed. http://ayende.com/4650/nosql-without-web-scale#comment34http://ayende.com/4650/nosql-without-web-scale#comment34Wed, 06 Oct 2010 21:06:32 GMT> you cannot combine data from different tables automatically That is what the map/reduce indexes are for. You can do that there. As for how this is implemented, we probably need to ask this in the mailing list. http://ayende.com/4650/nosql-without-web-scale#comment33http://ayende.com/4650/nosql-without-web-scale#comment33Wed, 06 Oct 2010 20:20:21 GMTFrank, No, not really. That is because the feature is so new, I haven't had the chance to blog about it :-) There are 3 ways to query RavenDB: * Indexes * Linear query (which is what I blogged about, table scan). * Auto Query - uses same syntax as usual indexing, but doesn't require an index, and very efficient. And actually, we DO support set based update/deletes :-) http://ayende.com/4650/nosql-without-web-scale#comment32http://ayende.com/4650/nosql-without-web-scale#comment32Wed, 06 Oct 2010 20:18:53 GMTgandjustas, I think it is easy to query the same table in different ways with raven (just add an index). I am in favor of that approach. But the problems start to arise (IMHO) when you need a new query that joins two tables. Then you are forced to do manual maintainance because indexes in raven do not support the join clause. In raven you can create an index that reformats an existing table but you cannot combine data from different tables automatically. I would be so happy if this feature was in the product. Manual maintainance of data structures _sucks_. Seemingly, the only support of this scenario to some extent is the "include" feature, but that always does a nested loops join. You cannot get hash or merge join with it. I believe that join indexes can be maintained efficiently as well because sql server can do it. Ayende, why do you not implement it? Do you want to set the right mindset in raven and discourage the use of joins? http://ayende.com/4650/nosql-without-web-scale#comment31http://ayende.com/4650/nosql-without-web-scale#comment31Wed, 06 Oct 2010 20:08:41 GMTAyende, yes you wrote about that, but you also said that a runtime query is the equivalent of a full table scan. This should be different to RDBMS'possiblity of reusing existing indices. btw, at some point all this stuff shouldn't be called NoSQL anymore. For all I know you could introduce a SQL parser to RavenDB to define your indices, and what then? Subsequent renaming of hundreds of blog posts! When you are young it's nice to differentiate yourself by saying what you are not, but that can't be the end of the road :) http://ayende.com/4650/nosql-without-web-scale#comment30http://ayende.com/4650/nosql-without-web-scale#comment30Wed, 06 Oct 2010 20:00:35 GMTJesus, That is a relatively new feature, but yes, it supports that. http://ayende.com/4650/nosql-without-web-scale#comment29http://ayende.com/4650/nosql-without-web-scale#comment29Wed, 06 Oct 2010 18:35:31 GMT@Ayende Does Raven DB support queries defined at run time? One advantage relational systems has over no SQL databases, is that you can create queries at run time, and they can create a query plan that can use existing indexes. It seems that with Raven DB you need a predefined index, without that intex it is is difficult to have a decent performant query defined at runtime. On the other hand, with no SQL databases, It seems to be a must to know query nedds up front. But query needs change over time. Furthermore, query defined at runtime cannot be predicted. I see no SQL databases can help in current days. But they cannot substitute RDBMS for now. http://ayende.com/4650/nosql-without-web-scale#comment28http://ayende.com/4650/nosql-without-web-scale#comment28Wed, 06 Oct 2010 18:03:19 GMTgandjustas, No, it doesn't. And RavenDB certainly supports linq. http://ayende.com/4650/nosql-without-web-scale#comment27http://ayende.com/4650/nosql-without-web-scale#comment27Wed, 06 Oct 2010 12:05:56 GMTCreating index on each query prevents dynamic query composition in application code, eg no Linq. http://ayende.com/4650/nosql-without-web-scale#comment26http://ayende.com/4650/nosql-without-web-scale#comment26Wed, 06 Oct 2010 12:04:02 GMT> "top sales artist" widget Create an index, query the index, done. http://ayende.com/4650/nosql-without-web-scale#comment25http://ayende.com/4650/nosql-without-web-scale#comment25Wed, 06 Oct 2010 11:28:50 GMTOh, MVC Music Store is a good example. It's not good data access with EF. There are lack of projections. In RavenDB version it's completly unextensible. What if I want to display "top sales artist" widget on pages? Or I need create Rating for Artists: user rates artists, rating affects catalog sorting etc? tobi mentioned "It is true that NoSql is less flexible in case of heavily interconnected data". But data can become "heavily interconnected" after applications shipped first time. It completly kills NoSQL for majority of applications. http://ayende.com/4650/nosql-without-web-scale#comment24http://ayende.com/4650/nosql-without-web-scale#comment24Wed, 06 Oct 2010 11:26:46 GMT@Peter Morlion, >You will have to solve problems where one aggregate needs to reference another, or part of another. It's will be a biggest problem, with most performance impact. http://ayende.com/4650/nosql-without-web-scale#comment23http://ayende.com/4650/nosql-without-web-scale#comment23Wed, 06 Oct 2010 11:12:09 GMTgandjustas, Not at all. For that matter, take a look at Raven's MVC Music Store example http://ayende.com/4650/nosql-without-web-scale#comment22http://ayende.com/4650/nosql-without-web-scale#comment22Wed, 06 Oct 2010 11:11:13 GMTMost difficult feature is implement ALL cases with good-enough performance without by-hand denormalization, aggregetion etc. http://ayende.com/4650/nosql-without-web-scale#comment21http://ayende.com/4650/nosql-without-web-scale#comment21Wed, 06 Oct 2010 11:09:26 GMT@gandjustas: I believe the NoSQL option has its benefits, but also its drawbacks. One of the big benefits for me is the fact that you no longer need complex mapping schemes (NHibernate) or ugly ActiveRecord-style code (or plain old SQL in your code, ugh). NoSQL does force you to construct your domain model nicely, with aggregate roots and such, but that can be seen as something positive. You will have to solve problems where one aggregate needs to reference another, or part of another. But even then, you get to focus on your domain, business and UI logic, which should be your core business. http://ayende.com/4650/nosql-without-web-scale#comment20http://ayende.com/4650/nosql-without-web-scale#comment20Wed, 06 Oct 2010 10:50:33 GMTFrank, If you have replication setup, you have automatic failover. http://ayende.com/4650/nosql-without-web-scale#comment19http://ayende.com/4650/nosql-without-web-scale#comment19Wed, 06 Oct 2010 10:34:01 GMTHow does the failover story read in RavenDB? http://ayende.com/4650/nosql-without-web-scale#comment18http://ayende.com/4650/nosql-without-web-scale#comment18Wed, 06 Oct 2010 10:32:06 GMT@gandjustas >>I'm already identified querying requirements in my first comment. But no one offered solution in NoSQL I'm asking you what you think is the most difficult feature so I can explain how you would achieve that particular functionality in detail. Providing a complete Stack Overflow solution is not the best of use of our time especially since the existing documentation I provided should give you a general idea on how you would use NoSQL to model the solution. http://ayende.com/4650/nosql-without-web-scale#comment17http://ayende.com/4650/nosql-without-web-scale#comment17Wed, 06 Oct 2010 10:01:06 GMT@gandjustas >>Can you tell what problems better modeled with document database? I would say anything that is non-relational would be a candidate. I actually view NoSQL dbs as a complimentary rather than a supplementary technology. It definitely isn't the right choice in all cases although there are clear scenarios where it holds advantages over a RDBMS - I list a few of them in my blog post here: [http://www.servicestack.net/mythz_blog/?p=129](http://www.servicestack.net/mythz_blog/?p=129) As is the case with any new technology there is sometimes a fear of the unknown when dealing with NoSQL db's, however I would approach NoSQL db's like learning a new language, once spending some time to get familiar with it you will learn different approaches to solving the same problem which will at the very least make you a better all-round developer. This will give you a better idea to assess where it makes sense to use it or not. The beauty of NoSQL is that most of them are free and are very easy to get started (typically just download the server and run). For a quick taste, Google App Engine actually provides a general purpose free hosting web development environment in Python or Java. It uses BigTable as its primary storage and I think you will be surprised how quick and frictionless it is to develop and deploy apps based on it. http://ayende.com/4650/nosql-without-web-scale#comment16http://ayende.com/4650/nosql-without-web-scale#comment16Wed, 06 Oct 2010 09:55:18 GMT@Demis Bellot: >IMHO the most unnatural part of building a db with NoSQL is to identify your querying requirements upfront... I'm already identified querying requirements in my first comment. But no one offered solution in NoSQL (I posted this requirement in other blogs and forums). http://ayende.com/4650/nosql-without-web-scale#comment15http://ayende.com/4650/nosql-without-web-scale#comment15Wed, 06 Oct 2010 09:49:47 GMTFrans, RavenDB is fully ACID. There is nothing in NoSQL that says tat you need to lose ACID. > Aggregates calculated on the fly really require a set-based language, not a document graph. Huh?! Let us talk about the Order Aggregate, what do you mean about it from there? http://ayende.com/4650/nosql-without-web-scale#comment14http://ayende.com/4650/nosql-without-web-scale#comment14Wed, 06 Oct 2010 09:40:47 GMT@Patrick Huizinga Can you tell what problems better modeled with document database? http://ayende.com/4650/nosql-without-web-scale#comment13http://ayende.com/4650/nosql-without-web-scale#comment13Wed, 06 Oct 2010 09:17:16 GMT@gandjustas I provided those examples because it is existing documentation available that closely matches what you want to achieve. You should be able to extrapolate based on those approaches to meet the other requirements. There is nothing inherently difficult about creating a StackOverflow using a NoSQL database. Of those features mentioned, what do you think would be the most difficult of those to maintain in a NoSQL db? IMHO the most unnatural part of building a db with NoSQL is to identify your querying requirements upfront so you can maintain indexes on them. You can always add the indexes after and doc db's like RavenDB and MongoDB also allow you to perform adhoc querying after the fact. So there is a little of *thinking different* and my first link on my previous comment should hopefully help with designing a NoSQL db from a RDBMS background. Included as test data for my Redis Admin UI demo I've imported the entire Northwind Relational Database you can see here: [www.servicestack.net/RedisAdminUI/AjaxClient/](http://www.servicestack.net/RedisAdminUI/AjaxClient/) From a list of POCO's importing the entire Northwind DB literally took around 11 lines of code (see bottom: [code.google.com/.../ServiceStackRedis](http://code.google.com/p/servicestack/wiki/ServiceStackRedis)) 1 LOC per table. http://ayende.com/4650/nosql-without-web-scale#comment12http://ayende.com/4650/nosql-without-web-scale#comment12Wed, 06 Oct 2010 09:07:43 GMTNoSQL isn't about scaling, it's about NoACID. [dbmsmusings.blogspot.com/.../...w-to-fix-them.html](http://dbmsmusings.blogspot.com/2010/08/problems-with-acid-and-how-to-fix-them.html) And before that guy is butchered to death, read his resume. What I also find funny is the remark "nosql db's are easier with aggregates", you really mean "storing readonly denormalized data", I think. Aggregates calculated on the fly really require a set-based language, not a document graph. Btw, it's not that RDBMS-s always are the best choice, it's just that claiming RDBMS-s are 'old news because something better came along' as NoSQL solved all the problems is simply naive. http://ayende.com/4650/nosql-without-web-scale#comment11http://ayende.com/4650/nosql-without-web-scale#comment11Wed, 06 Oct 2010 08:29:33 GMTFunny how a blog post about how NoSQL can also be a good choice besides scaling requirements turns into comments about 'scaling stackoverflow.com'. @qandjusta Just because you find a case where a document database is not the best fit for a problem from a modeling perspective doesn't invalidate the claim that some problems are better modeled with it. With the little I know, I would take a look into graph databases for highly connected data like stackoverflow. And there are requirements for consistency for a caching solution. You don't want the user not to see the update he just made. Remember that eventual consistency doesn't mean no consistency and results in delays at computer scale. For most scaling databases (be that NoSQL or YesSQL) the human reaction time is already considered as laughably slow. http://ayende.com/4650/nosql-without-web-scale#comment10http://ayende.com/4650/nosql-without-web-scale#comment10Wed, 06 Oct 2010 08:00:09 GMT@Demis Bellot Cache is not a primary data store. For caching there is no requirements for Consistency and Durability. http://ayende.com/4650/nosql-without-web-scale#comment9http://ayende.com/4650/nosql-without-web-scale#comment9Wed, 06 Oct 2010 02:45:37 GMT@Ayende Rahien Scaling is not necessary. Database with 1M Questions and 10M Answers, Comments and Votes will fit into 50Gb. It's not-so-large database. One server can easely handle this data. A want to see a NoSQL solution for StackOverflow cases. PS. StackOverflow has less than 1M Questions. http://ayende.com/4650/nosql-without-web-scale#comment8http://ayende.com/4650/nosql-without-web-scale#comment8Wed, 06 Oct 2010 02:35:21 GMT@gandjustas I have some documentation available on how you would build a simple blog using Redis available here: [code.google.com/.../DesigningNoSqlDatabase](http://code.google.com/p/servicestack/wiki/DesigningNoSqlDatabase) (There's also a refactored version that puts all redis access behind a repository pattern: [http://bit.ly/9niEHU](http://bit.ly/9niEHU)). This essentially mimics what ayende is doing with RavenDB in his series of blog posts here: [ayende.com/.../...al-modeling-anti-pattern-in.aspx](http://ayende.com/Blog/archive/2010/04/20/that-no-sql-thing-the-relational-modeling-anti-pattern-in.aspx) Using Redis Sets / Sorted Sets takes care of voting in a single, super-fast operation, in-fact Jeff Atwood (@CondingHorror of StackOverflow fame) has said that they are making use of Redis now in StackOverflow and all their StackExchange sites: [http://twitter.com/#](http://twitter.com/#)!/codinghorror/status/22417440038 At the moment it looks like its only used for their shared-caching solution although this is just another scenario where NoSQL db's provide superior solutions over RDBMS's. http://ayende.com/4650/nosql-without-web-scale#comment7http://ayende.com/4650/nosql-without-web-scale#comment7Tue, 05 Oct 2010 22:20:36 GMT