http://ayende.comAyende @ RahienCopyright (C) Ayende Rahien 2004 - 2021 (c) 202660Matthew, It goes something like this: documentStore.Commands.Put("foo", null, JObject.Parse("{ a: 1}"), new JObject(), null); http://ayende.com/4466/that-no-sql-thing-modeling-documents-in-a-document-database#comment68http://ayende.com/4466/that-no-sql-thing-modeling-documents-in-a-document-database#comment68Tue, 18 May 2010 00:52:07 GMTWonderful. Could you provide an example of creating document without an associated class? http://ayende.com/4466/that-no-sql-thing-modeling-documents-in-a-document-database#comment67http://ayende.com/4466/that-no-sql-thing-modeling-documents-in-a-document-database#comment67Tue, 18 May 2010 00:45:20 GMTMatthew, No, there is no requirement for a class. You can handle this totally dynamically. The .NET Client API comes in two forms, high level, which gives you an interface like NHibernate's session, entities, etc. That level requires you to work with classes, yes. But there is one layer down, where you can work directly against Json documents and do whatever your heart content to do with them http://ayende.com/4466/that-no-sql-thing-modeling-documents-in-a-document-database#comment66http://ayende.com/4466/that-no-sql-thing-modeling-documents-in-a-document-database#comment66Tue, 18 May 2010 00:39:10 GMTI'm thinking that RavenDB isn't for me. But I want to be sure. Are documents always tied to class objects? What if I don't have a class associated with my data? My data comes from CSV files, which don't all have the same structure, so I can't construct a class for them (they have similar structure, but some times different fields. I know that I can load the data into a datatable and put it into RavenDB from there. I've tried it, it works. However, I'm having trouble creating a query/index for the data that is contained in the datatable. Can I do this? I know that with MongoDB this is no problem. But I think Raven is a little different. http://ayende.com/4466/that-no-sql-thing-modeling-documents-in-a-document-database#comment65http://ayende.com/4466/that-no-sql-thing-modeling-documents-in-a-document-database#comment65Tue, 18 May 2010 00:36:05 GMT@Mike, @Medyum It really depends, the point of the layout in Post, is that is probably the most typical usage scenario. If you look at how the "related posts" are structured, it shows that information that is most needed is copied over. If you don't really *need* the user data in your document database (DDB), then just save the user-id from the rdbms. There's nothing stopping you from persisting some data in an RDBMS, and other data in the DDB. In fact, depending on your reporting needs, it may behoove you to have a regular process that normalizes some of the DDB data and exports it into an SQL-RDBMS. It just depends on your needs, and your usage scenario. I suggest the SQL-RDBMS export mainly for reporting needs, as the tooling available for reporting with SQL-RDBMS is so compelling compared to what is available to DDB, and the usage of SQL-RDBMS can support some of these types of scenarios better. There's nothing wrong with a hybrid approach, as long as it suits your needs, and is well defined/documented. http://ayende.com/4466/that-no-sql-thing-modeling-documents-in-a-document-database#comment64http://ayende.com/4466/that-no-sql-thing-modeling-documents-in-a-document-database#comment64Wed, 05 May 2010 01:34:18 GMTSure, what I mean is more in general, how would you handle a situation where you need (or want) to store some information in a relational database. For example, user accounts. Would you duplicate the user accounts in the documentdb? If not, how would you relate posts to users and preserve some kind of integrity. Thanks! http://ayende.com/4466/that-no-sql-thing-modeling-documents-in-a-document-database#comment63http://ayende.com/4466/that-no-sql-thing-modeling-documents-in-a-document-database#comment63Tue, 27 Apr 2010 08:12:08 GMTMike, I am not sure that I understand the question, can you expand? http://ayende.com/4466/that-no-sql-thing-modeling-documents-in-a-document-database#comment62http://ayende.com/4466/that-no-sql-thing-modeling-documents-in-a-document-database#comment62Mon, 26 Apr 2010 14:14:21 GMTThanks for writing all these posts, they have really helped me understand better. Would you consider explaining a situation where the users are stored in a relational database? Thanks! http://ayende.com/4466/that-no-sql-thing-modeling-documents-in-a-document-database#comment61http://ayende.com/4466/that-no-sql-thing-modeling-documents-in-a-document-database#comment61Mon, 26 Apr 2010 14:04:31 GMT@Jon Dokulil I understand you may be concerned about projection but personally I don't think this is a problem as there is no more complete LINQ support than LINQ 2 Objects. You can instantiate the entity with a single call then project it to a different POCO view using a combination of LINQ and a domain mapper tool like Auto Mapper to do all the heavy lifting for you: [http://automapper.codeplex.com/](http://automapper.codeplex.com/) I cache different intelligent views of the same data all the time where as soon as the master entity is modified all the dependent cached data views are invalidated. @Doron There are potentially many optimizations you can make, you can maintain a separate list of comments outside of the Post as Ayende has suggested. Though storing large blobs in a NoSQL datastore is not anywhere as inefficient as it is in an RDBMS as most of the time spent is CPU resources on the client de/serializing and network I/O throughput, this is in contrast with an RDBMS handling of large entities where most of the time is spent on the server processing and parsing the request extracting the data and updating internal data structures and indexes on disk. Most NoSQL datastores don't parse the payload at all (and some like Redis don't even need to touch disk), this is part of the reasons why NoSQL data stores are much quicker than RDBMS's for a lot of data access scenarios. If performance was a problem and *I had* to optimize my preferred approach would still be to maintain the 'latest page of comments' on the post entity while storing the rest of the comments in an external list. This way you can satisfy the data requirements for the post page view with a single request. http://ayende.com/4466/that-no-sql-thing-modeling-documents-in-a-document-database#comment60http://ayende.com/4466/that-no-sql-thing-modeling-documents-in-a-document-database#comment60Thu, 22 Apr 2010 22:58:51 GMTDoron, It might be, at that point, you can use partial updates to add a comment without retrieving the full document. http://ayende.com/4466/that-no-sql-thing-modeling-documents-in-a-document-database#comment59http://ayende.com/4466/that-no-sql-thing-modeling-documents-in-a-document-database#comment59Thu, 22 Apr 2010 19:07:10 GMTIsn't it rather inefficient that adding a comment requires retrieving the entire Post entity including all the other comments? http://ayende.com/4466/that-no-sql-thing-modeling-documents-in-a-document-database#comment58http://ayende.com/4466/that-no-sql-thing-modeling-documents-in-a-document-database#comment58Thu, 22 Apr 2010 17:23:22 GMTLike most, I'm coming from an RDBMS background. One thing that has been nice with using a relational DB for the app I'm currently working on is the ability to project the data in new ways, as new requirements/features come up. Over a year into the project we added a feature that was a completely new view of the data. The SQL queries that are needed to generate that new view are pretty hairy (they really make me appreciate LINQ to SQL), but they work and seems to be no problem for SQL Server. It seems like, using a DocDB approach we would have to write migration code more often to re-model our data in a way that suits both old and new features. I'm struggling to come up with a good example, but I'm imagining a case where our old assumption that a piece of data only exists within the context of some 'owning' object but then we want to add a feature where that is no longer a true assumption. Essentially, blogs are very well understood, your example is easy to follow. However, many projects revolve around domains that are not as well understood and have shifting requirements. A normalized RDB model seems to work pretty well for this. Do you think DocDBs are better suited for refactoring already well understood problems (perhaps to gain simplicity or for performance or scalability improvements), or would you feel comfortable going with a DocDB on a project where you expect shifting requirements down the road? http://ayende.com/4466/that-no-sql-thing-modeling-documents-in-a-document-database#comment57http://ayende.com/4466/that-no-sql-thing-modeling-documents-in-a-document-database#comment57Thu, 22 Apr 2010 14:30:53 GMTTom, That is another thing that you have to decide how to do, yes. And it affect how you design the documents. In general, most DocDBs knows either how to do a compare & swap or supports locking, so that isn't really a big problem. In Raven's case, you can ask it to perform an add to the comments collection, something which is safe to do concurrently. http://ayende.com/4466/that-no-sql-thing-modeling-documents-in-a-document-database#comment56http://ayende.com/4466/that-no-sql-thing-modeling-documents-in-a-document-database#comment56Thu, 22 Apr 2010 07:04:32 GMT@Tom Clarkson I see, you're explaining the pro's and con's of each approach. My preference would still be for comments to remain with the post, and personally judging by the quality of some of the comments on YouTube it wouldn't hurt to lose a few :) http://ayende.com/4466/that-no-sql-thing-modeling-documents-in-a-document-database#comment55http://ayende.com/4466/that-no-sql-thing-modeling-documents-in-a-document-database#comment55Thu, 22 Apr 2010 02:57:51 GMT@Demis Certainly it can be handled, though I prefer to avoid needing to handle locking if possible - If comment is a separate object, you don't have anything that needs to be modified by multiple users. Implementing in Redis I would keep the post itself as a single value (only editable by the author) and use a list (with atomic add operation) for the comments (either directly in the list or as seperate objects if they are to be editable). Implementation details aside, my point was mostly about the generic applicability of the model and what belongs in a best practice example where the required capabilities of the underlying platform are not specified. http://ayende.com/4466/that-no-sql-thing-modeling-documents-in-a-document-database#comment54http://ayende.com/4466/that-no-sql-thing-modeling-documents-in-a-document-database#comment54Thu, 22 Apr 2010 02:34:20 GMT@Tom Clarkson You can handle race collisions using 'Application level locks' (i.e. Redis's SETNX) where you only set if the value hasn't changed. Redis is going to introduce the notion of 'watched variables' within a transaction so if any of them has been updated since the transaction started the transaction is aborted. http://ayende.com/4466/that-no-sql-thing-modeling-documents-in-a-document-database#comment53http://ayende.com/4466/that-no-sql-thing-modeling-documents-in-a-document-database#comment53Thu, 22 Apr 2010 02:03:40 GMTStoring comments within the blog post is certainly simplest for reading, but introduces some risks on write - specifically what happens if two comments are added simultaneously. A specific document database implementation may understand how to apply two add operations, but that's not necessarily true when applied to a generic document store. http://ayende.com/4466/that-no-sql-thing-modeling-documents-in-a-document-database#comment52http://ayende.com/4466/that-no-sql-thing-modeling-documents-in-a-document-database#comment52Thu, 22 Apr 2010 01:48:09 GMT> I can put users on one machine and blogs on the other, and I don't have to have distributed transactions. Yeah that's the SOA-perspective part of the solution I liked. If I had to cross a service boundary I would only keep one of the references which in this case will be on the Blog. At the same time I would convert the ids to be urn's for any cross-service relationships. > I am not sure that I am following that The Linq statement is the "index definition" Yeah I just meant the manual work in creating LINQ statement for being able to query on the relationship. With my setup you could just skip the index definition and just do: redisBlogs.GetByIds(user.BlogIds); http://ayende.com/4466/that-no-sql-thing-modeling-documents-in-a-document-database#comment51http://ayende.com/4466/that-no-sql-thing-modeling-documents-in-a-document-database#comment51Wed, 21 Apr 2010 23:01:54 GMTDemis, > In my experience erring on predictable, convention-based solutions are more elegant, less error-prone, require less effort and allow you to provide richer API's that 'do more for free'. I get your point, and I agree. This is something that the client needs to worry about, and I am concentrating more on the server side for now. > the difference is that the relationship still exists in an external index The relationship exists in a once removed sort of way. I can put users on one machine and blogs on the other, and I don't have to have distributed transactions. > I also prefer a GetByIds() to a manually-crafted LINQ statement I am not sure that I am following that The Linq statement is the "index definition" http://ayende.com/4466/that-no-sql-thing-modeling-documents-in-a-document-database#comment50http://ayende.com/4466/that-no-sql-thing-modeling-documents-in-a-document-database#comment50Wed, 21 Apr 2010 22:37:54 GMT@Aydende > That is actually intentional. You either already know that (because you are following a strongly typed POCO) or you need to get that document. In my experience erring on predictable, convention-based solutions are more elegant, less error-prone, require less effort and allow you to provide richer API's that 'do more for free'. Intention over convention means that you have a lot more intelligence about the data embedded in the application itself and the 'data speaks less about itself'. I predict you will have a hard time providing a good generic GUI that lets you navigate the data store on its own (i.e. independently of any application logic). > Having the type encoded as the relationship isn't useful. I'm not sure I would agree with that, relationship encoded urns give you a chance to provide a generic solution to aggregate, fetch and instantiate the entites without internal knowledge of the data values. > That is pretty much the point, I don't see a direct association between users & blogs as a good thing. I _want_ you to search on them It is a unique perspective on one hand I like it from a SOA perspective where one entity is the master reference and 'owns the relationship', OTOH the difference is that the relationship still exists in an external index, 'its just not visible'. I also prefer a GetByIds() to a manually-crafted LINQ statement to achieve the same result (although LINQ in DHT datastore is cool :). http://ayende.com/4466/that-no-sql-thing-modeling-documents-in-a-document-database#comment49http://ayende.com/4466/that-no-sql-thing-modeling-documents-in-a-document-database#comment49Wed, 21 Apr 2010 22:14:52 GMTRyan, Can you explain where you lost the thread of the conversation? http://ayende.com/4466/that-no-sql-thing-modeling-documents-in-a-document-database#comment48http://ayende.com/4466/that-no-sql-thing-modeling-documents-in-a-document-database#comment48Wed, 21 Apr 2010 21:43:39 GMTDemis, > - the value on it's own 'blogs/1' is not distinguishable from a normal text value. I don't worry about it. That is something that the client can decide on, not something that the database enforce. > - 'blogs' is not a direct correlation to its POCO type 'Blog' so you will need to maintain the link via some external configuration That is actually intentional. You either already know that (because you are following a strongly typed POCO) or you need to get that document. Having the type encoded as the relationship isn't useful. > - by concatenating it with a string 'blogs/1' it requires effort to get the strongly-typed id of '1' which means being able to it automatically incremented is not possible. Raven supports this, actually. You can ask it to generate a key with a given prefix, which gives us the nice keys for the documents. Admittedly, I implemented that feature after I wrong the post, because I had the same objection :-) > searching is a very nice to have in a lot of places I just don't think it belongs in the construction of your domain model That is pretty much the point, I don't see a direct association between users & blogs as a good thing. I _want_ you to search on them http://ayende.com/4466/that-no-sql-thing-modeling-documents-in-a-document-database#comment47http://ayende.com/4466/that-no-sql-thing-modeling-documents-in-a-document-database#comment47Wed, 21 Apr 2010 21:43:16 GMTI'm pretty comfortable with my intellect...but man this discussion makes me feel dumb. It's not 'clicking' at all. http://ayende.com/4466/that-no-sql-thing-modeling-documents-in-a-document-database#comment46http://ayende.com/4466/that-no-sql-thing-modeling-documents-in-a-document-database#comment46Wed, 21 Apr 2010 21:33:46 GMT@Ayende: I've noticed you've taking a rest-like scheme to storing entities e.g. 'users/ayende' and 'blogs/1', although semantically correct, I think you may run into unnecessary problems as: - the value on it's own 'blogs/1' is not distinguishable from a normal text value. - 'blogs' is not a direct correlation to its POCO type 'Blog' so you will need to maintain the link via some external configuration - by concatenating it with a string 'blogs/1' it requires effort to get the strongly-typed id of '1' which means being able to it automatically incremented is not possible. My preference is to rely on convention, e.g. the 'User' model would still to be to store the strongly typed id in the model which would make it stricter and allows the NoSQL database to increment the 'id sequence' to guarantee uniqueness. In order to keep each entity unique within the same datastore I use a predictable urn as 'entity id' e.g: urn:User:1 => { Id = 1, Name = "Ayende", BlogIds = [1,2,3] } urn:Blog:1 => { Id = 1, UserId = 1, } In this way, no other configuration/manual intervention is required and I can get automatic sequencing. The URN's can now be used outside of the application to identify an entity in a universally identifiable in a standardized format. The Urn also contains the exact type of the entity it identifies which is useful when you want to generically handle a bag of mixed URNs generically. You also have decided not to have the User dual-referencing it's blog posts. Which means you are forced to 'search' for your related entities through an external index. Personally I don't like this idea, searching is a very nice to have in a lot of places I just don't think it belongs in the construction of your domain model. http://ayende.com/4466/that-no-sql-thing-modeling-documents-in-a-document-database#comment45http://ayende.com/4466/that-no-sql-thing-modeling-documents-in-a-document-database#comment45Wed, 21 Apr 2010 20:48:06 GMTRalf, I am using this to demonstrate things, not as the real implementation. Raven supports additional metadata for a document, so the client API can do something like: { "@metadata": { "type": " Full .Net Type Name" }, // other content } http://ayende.com/4466/that-no-sql-thing-modeling-documents-in-a-document-database#comment44http://ayende.com/4466/that-no-sql-thing-modeling-documents-in-a-document-database#comment44Wed, 21 Apr 2010 19:57:39 GMTRafal, Yes, I built a relatively large system on top of a KeyValue store (Rhino DHT). It worked very well, and it simplified what we needed to do drastically. It also gave _very_ low latency overall. http://ayende.com/4466/that-no-sql-thing-modeling-documents-in-a-document-database#comment43http://ayende.com/4466/that-no-sql-thing-modeling-documents-in-a-document-database#comment43Wed, 21 Apr 2010 19:55:41 GMTRalf, > new Blog(..., new[] { new Comment(...), ... }) There is no reason, both representations are valid. http://ayende.com/4466/that-no-sql-thing-modeling-documents-in-a-document-database#comment42http://ayende.com/4466/that-no-sql-thing-modeling-documents-in-a-document-database#comment42Wed, 21 Apr 2010 19:54:51 GMTBarry, Yes, an ASP.Net service is very possible, and should be very easy to do. http://ayende.com/4466/that-no-sql-thing-modeling-documents-in-a-document-database#comment41http://ayende.com/4466/that-no-sql-thing-modeling-documents-in-a-document-database#comment41Wed, 21 Apr 2010 19:53:39 GMTOn the type field in the JSON objects: During serialization you introduce a new field in addition to an objects properties. "type" stores the name of the serialized type. Why don´t you prefix it somehow to distinguish it from the object´s properties? What, if an object contains a property called type? I´d either use well known GUIDs for RavenDB field names or at least prefix them with something like "ranvendb_", e.g. "ravendb_type". http://ayende.com/4466/that-no-sql-thing-modeling-documents-in-a-document-database#comment40http://ayende.com/4466/that-no-sql-thing-modeling-documents-in-a-document-database#comment40Wed, 21 Apr 2010 19:40:44 GMTAyende, have you done any real-world, production software based on a NoSQL document database? I'm asking because you are taking an expert's standpoint advicing on software architecture, therefore I'd like to know how successful was this architecture in your case. http://ayende.com/4466/that-no-sql-thing-modeling-documents-in-a-document-database#comment39http://ayende.com/4466/that-no-sql-thing-modeling-documents-in-a-document-database#comment39Wed, 21 Apr 2010 17:51:48 GMT