http://ayende.comAyende @ RahienCopyright (C) Ayende Rahien 2004 - 2021 (c) 202660Re: Searching: Could you compute hashes for documents as they were added to the db and return the hash value to the calling app to use later for retrieval? You might also be able to use the hash to search for identical documents with different Id's. Allowing calling applications to add tags to a document would be useful for searching also. http://ayende.com/3897/designing-a-document-database#comment13http://ayende.com/3897/designing-a-document-database#comment13Tue, 10 Mar 2009 18:18:03 GMTMy team and I developed a "document repository" late last year. We had different constraints, but some of the challenges for us were: * delivering documents to the web when they are stored in a secure repository inside our firewall * deciding sql blobs vs filesystem (we chose filesystem, even though we could have done a hybrid, i.e. SQL 2008) * workflow, i.e. allowing different people to see docs at different places in a workflow Other challenges which I doubt you will have, related to being able to deal with groups of related docs: * determining duplicates (what makes a document an update vs a new doc) * versioning of groups of documents vs versioning individual docs http://ayende.com/3897/designing-a-document-database#comment12http://ayende.com/3897/designing-a-document-database#comment12Mon, 09 Mar 2009 18:25:44 GMTLooking forward to these posts, Ayende. Good outline. http://ayende.com/3897/designing-a-document-database#comment11http://ayende.com/3897/designing-a-document-database#comment11Mon, 09 Mar 2009 16:45:45 GMTLooking forward to your next posts on this topic. Right now I'm evaluating the same but from perspective of offline client. We are building distributed clients that need to pull some data (mostly read-only) and image attachments. I was thinking about using the System.IO.Packaging format to store this as documents on the client (like Office docs). Do you think it's related? If so please consider it in your series as well. http://ayende.com/3897/designing-a-document-database#comment10http://ayende.com/3897/designing-a-document-database#comment10Mon, 09 Mar 2009 16:15:06 GMTMarco, I don't think that I am going to handle archiving. You can run a view and do it as a separate process, outside of the DB. Working offline, that seems like a special case of replication. In this case, we replicate to the master, and we have to do it in an async manner http://ayende.com/3897/designing-a-document-database#comment9http://ayende.com/3897/designing-a-document-database#comment9Mon, 09 Mar 2009 14:15:35 GMTAnother one: - working offline . local (partial) backup and commit changes/new items to the server http://ayende.com/3897/designing-a-document-database#comment8http://ayende.com/3897/designing-a-document-database#comment8Mon, 09 Mar 2009 12:16:40 GMT-- archiving - can you explain more? Old documents that are never touched anymore can be moved to an archive database. Most DMS systems supports some kind of document lifecycle workflow / records management (for example document with type x (some metadata field) should be saved 10 years, after that it should be destroyed) (see [http://en.wikipedia.org/wiki/Records_management](http://en.wikipedia.org/wiki/Records_management)) http://ayende.com/3897/designing-a-document-database#comment7http://ayende.com/3897/designing-a-document-database#comment7Mon, 09 Mar 2009 12:11:11 GMTMarco, meta data - I am addressing that security - haven't thought about that, will need to address this versioning - I am addressing that searching - I am addressing that archiving - can you explain more? http://ayende.com/3897/designing-a-document-database#comment6http://ayende.com/3897/designing-a-document-database#comment6Mon, 09 Mar 2009 10:08:39 GMTUriel, The problem with layer the DB on top of the DHT is that you need to maintain additional data that the actual DHT doesn't let you. Lists of documents to be processed, for example. Rafal, Take a look at CouchDB, that is the source of much of the design. http://ayende.com/3897/designing-a-document-database#comment5http://ayende.com/3897/designing-a-document-database#comment5Mon, 09 Mar 2009 10:07:24 GMTAnd what about: - the meta-data on top of documents? - security - versioning - searching (full-text and on meta-data) - archiving (document lifecycle) http://ayende.com/3897/designing-a-document-database#comment4http://ayende.com/3897/designing-a-document-database#comment4Mon, 09 Mar 2009 08:57:33 GMTVery nice, but for a starter I'd like to see some background: - what is a document - how does it differ from an attachment - what is a view - how does document lifecycle look like - what will be the intended use of the software you are designing http://ayende.com/3897/designing-a-document-database#comment3http://ayende.com/3897/designing-a-document-database#comment3Mon, 09 Mar 2009 08:49:35 GMTa really simple design will be to build that db on top of a persistent DHT so you get replication and scalability right away,or you could build it on top of some DFS. for the views you can make some incremental background process that updated views from documents,or shift the cost of updating views to the queries i think it is called cracking databases i think. http://ayende.com/3897/designing-a-document-database#comment2http://ayende.com/3897/designing-a-document-database#comment2Mon, 09 Mar 2009 07:11:03 GMTIt's called Sharepoint :) http://ayende.com/3897/designing-a-document-database#comment1http://ayende.com/3897/designing-a-document-database#comment1Mon, 09 Mar 2009 01:53:36 GMT