http://ayende.comAyende @ RahienCopyright (C) Ayende Rahien 2004 - 2021 (c) 202660You can quote Codd up the yin-yang, but it really doesn't change the fact that when 99.9% of the planet talks about "relational" DBs they aren't referring to INSERT and SELECT statements, they are talking normalization, they're talking about Foreign Keys, etc" Perhaps the problem that I am so upset about IS the fact that a large percentage of the programming population thinks "relational" is something that it is not, and is now rehashing old technologies like key-value stores and doc db's like they invented something new. "he very word "relational" in the "relational data model" is the key - it indicates to the relations among various entities using foreign keys etc. Use of SELCT INSERT has got nothing to to do with being relational." See here is my point, please read some Codd and understand what a relation is before saying something like this. You wonder what I am getting upset about when I get called disingenuous for simply wanting people to know the relational model before they bash it. "It does seem rooted in the fact that because you can make MSSQL/RDBMS act like a document database that there is no reason to go the NoSQL solution specifically catered for the task." I never said the is no reason to use a NoSQL db. I just wanted to give some perspective on the past, because most of the arguments we hashed out 30 years ago, and all the sudden no one can remember what the relational model is. http://ayende.com/4484/document-databases-are-not-relational#comment56http://ayende.com/4484/document-databases-are-not-relational#comment56Mon, 10 May 2010 14:34:31 GMT@Justin I am really not sure what you are arguing about any more? It does seem rooted in the fact that because you can make MSSQL/RDBMS act like a document database that there is no reason to go the NoSQL solution specifically catered for the task. Sure as a persistence layer you can potentially map any data structure to a relational model, it doesn't make it a good idea. The best solution is one that 'works the best' not 'any that works with what I know'. For some insight on the decision making process that leads towards adoption of a NoSQL solution over an RDBMS one, check out this post from digg's architect on the subject, here are a couple of choice quotes: [stu.mp/.../...l-vs-rdbms-let-the-flames-begin.html](http://stu.mp/2010/03/nosql-vs-rdbms-let-the-flames-begin.html) "Do you honestly think that the PhDs at Google, Amazon, Twitter, Digg, and Facebook created Cassandra, BigTable, Dynamo, etc. when they could have just used a RDBMS instead?" " NoSQL solutions allow us to serve absurd amounts of data for a really, really low price. I’m happy to put my $/write, $/read, and $/GB numbers for my NoSQL setup against anyone’s RDBMS numbers. We’re not nearly as dumb as everyone thinks we are; I promise." http://ayende.com/4484/document-databases-are-not-relational#comment55http://ayende.com/4484/document-databases-are-not-relational#comment55Sun, 09 May 2010 16:37:56 GMT@Justin: chill man!! the very word "relational" in the "relational data model" is the key - it indicates to the relations among various entities using foreign keys etc. Use of SELCT INSERT has got nothing to to do with being relational. Having/creating a bunch of tables in a SQL/Oracle/MySql database doesn't make it a relational database. http://ayende.com/4484/document-databases-are-not-relational#comment54http://ayende.com/4484/document-databases-are-not-relational#comment54Sun, 09 May 2010 15:33:20 GMTJustin, At the end of the day, you're being very disingenuous, and quite frankly I have no idea why you are so angry about some guys blog just because you define every database that can be used in a "relational" manner is automatically considered "relational". You can quote Codd up the yin-yang, but it really doesn't change the fact that when 99.9% of the planet talks about "relational" DBs they aren't referring to INSERT and SELECT statements, they are talking normalization, they're talking about Foreign Keys, etc. Think of it this way, I have 4 wheel drive on my SUV, it has "4WD" right on the decal, and in the sales literature it says that it has that capabilities. But I choose to use it in 2WD mode when I drive to work. You could argue that my Jeep is 4WD (and it is), but I don't use it that way (most of the time anyways). Not a perfect analogy, but good enough. Chill out and relax just a bit, you're freaking out over what is basically syntax, and it's making you look bad. http://ayende.com/4484/document-databases-are-not-relational#comment53http://ayende.com/4484/document-databases-are-not-relational#comment53Fri, 07 May 2010 19:19:43 GMTI thought you did know what the relational model is, but when you say nonsense like "Just using INSERT or SELECT statement doesn't mean that I am working in a relational form." then I start to wonder. The SQL language including inserts and selects IS First order predicate logic, table contents IS a relation, and yet here you are saying it's not! "No, it isn't. Insert and Select and just readable way to do INSERT("ayende","ayende@ayende.com", 38840) and SELECT("ayende")" That is the definition of Codd's 7th rule, did you forget this too? Do you need a refresher on how to tell if something is a relational db: [http://en.wikipedia.org/wiki/Codd](http://en.wikipedia.org/wiki/Codd)'s_12_rules ? Friend Feed built a key-value system ON TOP of a set of tables, they update and retrieve data with SQL. They even describe their table structure complete with primary and foreign keys! Are you seriously going to tell me Friend Feed is some how using MySQL in "non-relational mode"? Friend Feeds MySQL db still meets all of Codd's 12 rules. If I run Windows in a VM on my Mac, I am still running a Mac. If I build a Key-Value store on top of a relational db I am still using a relational db! http://ayende.com/4484/document-databases-are-not-relational#comment52http://ayende.com/4484/document-databases-are-not-relational#comment52Fri, 07 May 2010 19:08:44 GMTRay, Yes. Commercially speaking, I built a fairly large system on top of a DHT, in another project we had external indexes for most of the queries. http://ayende.com/4484/document-databases-are-not-relational#comment51http://ayende.com/4484/document-databases-are-not-relational#comment51Fri, 07 May 2010 14:44:28 GMTOren, on paper it sounds very good but the question is did you actually have experience with No SQL in a real production environment? No offense intended, I'm just curious. http://ayende.com/4484/document-databases-are-not-relational#comment50http://ayende.com/4484/document-databases-are-not-relational#comment50Fri, 07 May 2010 14:26:56 GMTFizz, That insert command is "store a tuple, keyed by the first item". As for what Doc DBs are good for, read my NoSQL category, it is all there. http://ayende.com/4484/document-databases-are-not-relational#comment49http://ayende.com/4484/document-databases-are-not-relational#comment49Fri, 07 May 2010 09:58:41 GMTINSERT("ayende","ayende@ayende.com", 38840) Is this a a document database command? It is not in the form K -> V, as it has three parameters. Can you please tell me, beyond storing non-related property bags, document DBs are useless? http://ayende.com/4484/document-databases-are-not-relational#comment48http://ayende.com/4484/document-databases-are-not-relational#comment48Fri, 07 May 2010 03:27:15 GMT> Using insert and select against tables IS the definition of the Relational Model. No, it isn't. Insert and Select and just readable way to do INSERT("ayende","ayende@ayende.com", 38840) and SELECT("ayende") Relational model include things like predicate logic, joins, fk, relations, etc. When I am talking about using something like MySql as a key/value store, I am referring to only issuing queries by id, a very common approach. That isn't relational. As a good example of that, see: [bret.appspot.com/entry/how-friendfeed-uses-mysql](http://bret.appspot.com/entry/how-friendfeed-uses-mysql) On a general note, I would really appreciate if you won't assume I am not aware of what I am talking about. http://ayende.com/4484/document-databases-are-not-relational#comment47http://ayende.com/4484/document-databases-are-not-relational#comment47Thu, 06 May 2010 23:46:45 GMT"Just using INSERT or SELECT statement doesn't mean that I am working in a relational form. MySQL is actually heavily utilized as a key/value store." Using insert and select against tables IS the definition of the Relational Model. The core definition:"Its central idea is to describe a database as a collection of predicates over a finite set of predicate variables, describing constraints on the possible values and combinations of values." How SQL the language and a DB that uses SQL is the the relational model:" A table in an SQL database schema corresponds to a predicate variable; the contents of a table to a relation; key constraints, other constraints, and SQL queries correspond to predicates." I would suggest you read some Codd(The guy who defined the relational model): [www.seas.upenn.edu/~zives/03f/cis550/codd.pdf](http://www.seas.upenn.edu/~zives/03f/cis550/codd.pdf) You can build a key-value store on top of the relational model in MySQL, and yes many have done so. Why, because MySQL performed well even compared to native key-value implementations. Just like I showed how to make a doc db using MSSQL's xml data type, MSSQL is still relational, but I am hiding it's relational model behind a document based application layer. An OO database can also be built on a Realtional Database, you should know since NHibernate does just that. Is MSSQL in non-relational mode when I use NHibernate against it? As far I know Google has never said they are using MySQL any other way then just straight relational with some sharding techniques, so your just grasping any when you say they use it in a key-value store manner anyway. Jason Slocomb, "Justin you are trolling. Just because Google or Yahoo hasn't migrated an existing production database consisting of 2 Petabytes of data supporting hundreds of thousands of users to a DocumentDB is not an indictment against the NOSQL movement or Raven." Except Yahoo's db wasn't a production db they migrated TO the SQL db from NoSQL, here straight from Waqar Hasan, vice president of engineering in Yahoo's data group: "Hasan joined Yahoo more than three years ago. At the time, Yahoo already had huge non-SQL databases storing hundreds of terabytes of data. Problem was, the data was in the form of large collections of compressed files that could be accessed only by writing programs in a language such as C++, rather than more easily and quickly via SQL commands, he said." and: "The top layer remains PostgreSQL, however, so that Yahoo can use the many off-the-shelf tools available for it." So if getting the facts straight on the largest most heavlily used DB's in the world being SQL and not NoSQL is trolling, I am truly sorry. http://ayende.com/4484/document-databases-are-not-relational#comment46http://ayende.com/4484/document-databases-are-not-relational#comment46Thu, 06 May 2010 17:18:31 GMTI'm on this with Fizz. Highly doubt that document DB's will be used extensively any time soon. They only greatly benefit from map-reduce pattern, and map-reduce is pointless if you don't have a cluster (map - split job, reduce - flatten results into one) and use cases when you need to do a lot of such processing on your data store... Good for Amazon, not so good for most of enterprise applications. http://ayende.com/4484/document-databases-are-not-relational#comment45http://ayende.com/4484/document-databases-are-not-relational#comment45Thu, 06 May 2010 11:26:42 GMTThe process is called "Denormalisation" i.e. you work out how to optimally store the data then make tradeoffs, not start with a tradeoff In reverse, storing data denormalised leads to storage bloat and data inconsistancies? On performance then, try removing every post and comment by a user Beyond storing non-related property bags, document DBs are useless? The Google search engine, is not ACID complient, it will never be consistant, the same query at the same point in time for 2 users is not garrenteed to return the same result. http://ayende.com/4484/document-databases-are-not-relational#comment44http://ayende.com/4484/document-databases-are-not-relational#comment44Thu, 06 May 2010 02:17:16 GMT> Do you understand the "only store once" principle of a relational DB Do you understand that in many high usage scenarios the "only store once" principle leads to horrible performance? http://ayende.com/4484/document-databases-are-not-relational#comment43http://ayende.com/4484/document-databases-are-not-relational#comment43Thu, 06 May 2010 01:55:24 GMTDo you know what BCNF is? Do you understand the "only store once" principle of a relational DB? Do you know what a semantic (o,p,v) store is? Beyond storing non-related property bags, document DBs are useless. http://ayende.com/4484/document-databases-are-not-relational#comment42http://ayende.com/4484/document-databases-are-not-relational#comment42Wed, 05 May 2010 22:33:45 GMTJustin you are trolling. Just because Google or Yahoo hasn't migrated an existing production database consisting of 2 Petabytes of data supporting hundreds of thousands of users to a DocumentDB is not an indictment against the NOSQL movement or Raven. http://ayende.com/4484/document-databases-are-not-relational#comment41http://ayende.com/4484/document-databases-are-not-relational#comment41Wed, 05 May 2010 22:28:41 GMTJustin, Just using INSERT or SELECT statement doesn't mean that I am working in a relational form. MySQL is actually heavily utilized as a key/value store. I suggest you will read this, it might help explain the basis of the issue: [www.25hoursaday.com/.../...LargeScaleWebsites.aspx](http://www.25hoursaday.com/weblog/2010/03/10/BuildingScalableDatabasesAreRelationalDatabasesCompatibleWithLargeScaleWebsites.aspx) http://ayende.com/4484/document-databases-are-not-relational#comment40http://ayende.com/4484/document-databases-are-not-relational#comment40Wed, 05 May 2010 19:58:20 GMTBoth MySQL and PostgresSQL are SQL databases, they store data in sets called tables and retrieve sets using SQL, how exactly do you run MySQL in non-relational mode? You may be able to forgo some ACID compliance in MySQL(just like using read uncomitted in MSSQL) by using one of the various storage engines, but you are always dealing with it relationally, as in tables and joins. The modified Postgres Yahoo uses, changed the storage engine to be column based from row based, this however doesn't change the fact that they are still using SQL(with joins, etc.) and dealing with sets, it is hidden under the covers, like all DBMS's storage engines. They specifically noted the reason the choose to modify Postgres instead of using a NoSQL solutions was because existing tools and applications could run with little change. http://ayende.com/4484/document-databases-are-not-relational#comment39http://ayende.com/4484/document-databases-are-not-relational#comment39Wed, 05 May 2010 19:50:32 GMT> Except the largest databases in the world are running relational. What on Earth gave you that impression? Google ads run on MySql, but there is no information on whatever it is running on MySQL using relational mode, or as a record store or even key/value store. Frankly, I am leaning toward the later, rather then the former. At any rate, I think that I can safely guess that the data is heavily sharded. As for Yahoo, they are running on Postgres, but not as a RDBMS, they are running a modified version that runs as a column database. Quite a different beast. http://ayende.com/4484/document-databases-are-not-relational#comment38http://ayende.com/4484/document-databases-are-not-relational#comment38Wed, 05 May 2010 18:27:32 GMT"Of course, as I said, you need to build that on top of MSSQL, turning your code into a doc db that uses MSSQL as a storage." No I am using SQL extended with XQuery, it is part of the DBMS, not something I am building on top of it. That would be like saying storing rows in MSSQL using SQL is turing my code into a relational db that uses MSSQL as storage. "Raven store its information in a ISAM DB internally. Storage for docs is just that, simple storage. A database tend to do more than just store bits." Storage isn't very useful with out a way to retrieve the data later right? So it's not just storing bits, not even the filesystem is just storing bits. Every datastore has various helpers for retrieving those bits later and making sure they are consistent. "Most file systems tend to fall over and die when you store millions of small files on them (oh, they work, but veeeeerrrrryyyy sloooooooooowwwly" Depends on the filesystem and how you store the data in it, your statement is a gross generalization. I bet I can store data in Raven in such a way to make it fall over and die as well. You write your serializer to match the capability of the data store, the more tools the data store provide the less you write in your serializer yourself. "You seem very firm in your opinions." I am very open to new DB technologies, relational has started to stagnated and is some what related to mindshare going to NoSQL. It is simply annoying to see the same arguments rehashed every 10 years, with the "Next Big Thing", Seems liek most of the arguments for NoSQL forget why relational won out before. OK I'll shut up now. http://ayende.com/4484/document-databases-are-not-relational#comment37http://ayende.com/4484/document-databases-are-not-relational#comment37Wed, 05 May 2010 17:30:23 GMT" think some of you are failing to realize that cow is already out of the barn with regards to NOSQL. RDBMS's are not always well suited for the kinds of challenges Amazon, Facebook, etc. are dealing with. The big boys are already rolling their own, ie; voldemort, Bigtable, etc." Except the largest databases in the world are running relational. Google ad words runs on Mysql, not Big table. Yahoo did have the record for the largest db in the world(2 petabytes) running on Postgres. Document db's are not new they predate relational, everyone just seems to have forgotten what happend a few decades ago, the industry is cyclic. Same things where said when OO db's became popular, and then XML db's, now we have NoSQL(which is just a rehash of IMS from the 1960's). http://ayende.com/4484/document-databases-are-not-relational#comment36http://ayende.com/4484/document-databases-are-not-relational#comment36Wed, 05 May 2010 16:46:18 GMTJustin, > You can do all those things with the XML datatype in MSSQL. Of course, as I said, you need to build that on top of MSSQL, turning your code into a doc db that uses MSSQL as a storage. Raven store its information in a ISAM DB internally. Storage for docs is just that, simple storage. A database tend to do more than just store bits. > just store them as blobs in a file system Most file systems tend to fall over and die when you store millions of small files on them (oh, they work, but veeeeerrrrryyyy sloooooooooowwwly But I think that we have better cut the argument right now. You seem very firm in your opinions. I suggest taking a look at my posts series on NoSQL to see why you might want to use such a database. http://ayende.com/4484/document-databases-are-not-relational#comment35http://ayende.com/4484/document-databases-are-not-relational#comment35Wed, 05 May 2010 16:34:58 GMT"Storage for Doc DB can be anything, flat files, SQL Server, etc. Hell, you can say that the file system is a doc db according to your logic." "A doc db provide more than just binary persistence. It provide querying capabilities, transactions, aggregation (usually in the form of map/reduce), conflict detection & resolution, etc." You can do all those things with the XML datatype in MSSQL. What does Raven store it's documents in, a file right? What is the difference between storage for documents and a document db? All DB's are is an abstraction layer over data that provides useful tools for update/retrieval right? Raven meets this definition, so MSSQL using a xml datatype. A filesystem on most OS's is a hierarchal db, some OS's even give you a way to index and query the filesystem. Some OS's file systems are closer to full DBMS, ironic that we build most of our DB's(Relational and Document) on top of a Hierarchal db. Now lets build an OO database on top of a document database on top of a hierarchal DB, heck we've been doing that with ORM's for years... "Not sure that I understand what you mean by "at least the document can be keyed", can you explain a bit more?" I have the option of using the document id as a key, I can make relational tables in my document db with foreign keys to the documents if I need to, you don't have that option in a pure document db. My point is a full featured relational database with hierarchal data types can do what document database can, plus all the relational features if you need them. "As for full object graph (including circular references), that is a pretty simple problem to solve. Hell, it is a first year problem when you use JSON or XML. " So why use foreign keys or joins in a relation db with an ORM? Just store object ID's and let the application layer enforce integrity right? What happens when I delete a object referenced by id in the document, oh now my app code has to go make sure I can delete, oops I forgot one place I stored the id, now we a reference to nothing the the app code has to handle. Why store objects in a document db at all, just store them as blobs in a file system. Hell, it is a first year problem when you use Files for serialization, right? http://ayende.com/4484/document-databases-are-not-relational#comment34http://ayende.com/4484/document-databases-are-not-relational#comment34Wed, 05 May 2010 16:27:01 GMTI think some of you are failing to realize that cow is already out of the barn with regards to NOSQL. RDBMS's are not always well suited for the kinds of challenges Amazon, Facebook, etc. are dealing with. The big boys are already rolling their own, ie; voldemort, Bigtable, etc. While the example Oren is using (blog) can be done well with a traditional Rdb, it's also an example everyone can wrap their mind around and discuss. That said, this contrived example should not be used as measuring stick for whether or not the DocDB paradigm has any validity. http://ayende.com/4484/document-databases-are-not-relational#comment33http://ayende.com/4484/document-databases-are-not-relational#comment33Wed, 05 May 2010 16:12:02 GMTRay, You missed the point. You won't embed the entire user information inside post. Indeed, that makes no sense. In the case of a Post and a User, we will embed the user's id, and we might embed the user's name, so we won't need to look it up in the user document just for display purposes. But we would certainly not replicate the entire user's document. If I really needed that much of the user's data, I would simply refer to the user's document http://ayende.com/4484/document-databases-are-not-relational#comment32http://ayende.com/4484/document-databases-are-not-relational#comment32Wed, 05 May 2010 15:25:28 GMTthe links to pictures are broken http://ayende.com/4484/document-databases-are-not-relational#comment31http://ayende.com/4484/document-databases-are-not-relational#comment31Wed, 05 May 2010 14:27:14 GMTAdditionally: if certain author is awarded with a new badge, how can we be sure that ALL his posts will be updated? i.e. how to ensure data consistency effectively? http://ayende.com/4484/document-databases-are-not-relational#comment30http://ayende.com/4484/document-databases-are-not-relational#comment30Wed, 05 May 2010 12:10:09 GMTDocument DB's are certainly interesting but honestly I still fail to see that many cases when it will have an edge over RDB's... For example each post will have an author and each author will have some statistics (badges, title, ect) that can be changed any time and certain data that will be changed everytime he posts something (number of posts). How do we handle that without hassle of going through all posts, selecting only posts with this particular author (kk, maybe fast through the index but still it will load the server), updating all of them and saving them back into DB... Sorry if my question sounds ignorant.. I really need to dig more into this subject when I get more time. http://ayende.com/4484/document-databases-are-not-relational#comment29http://ayende.com/4484/document-databases-are-not-relational#comment29Wed, 05 May 2010 12:01:36 GMTJdn, Oh, now it makes sense, I don't store it. I query that information from an index sitting on top of the posts.Tags collection http://ayende.com/4484/document-databases-are-not-relational#comment28http://ayende.com/4484/document-databases-are-not-relational#comment28Wed, 05 May 2010 04:44:44 GMT@jdn I think you would just need to define an index that pulls out all the tags applied to all the posts in the doc db. I believe you could also do it with a simple map/reduce and get how frequently each tag gets used as well. Something like: // Map from post in docs from tag in post.tags select new { tag, count = 1 }; // Reduce from pair in results group pair by pair.tag into g select new { pair.key, count = g.Sum(x=>x.count }; http://ayende.com/4484/document-databases-are-not-relational#comment27http://ayende.com/4484/document-databases-are-not-relational#comment27Wed, 05 May 2010 00:58:18 GMT