Ayende @ Rahien

evereq commented on Abstracting the persistence medium isn’t going to let you switch persistence abstractions

Tue, 03 Aug 2010 20:18:49 GMT

c) Here I want to go to our example with concrete methods, but my "energy' become too low :) So I will be brief: - it's completely possible to implement * GetPost(id) * GetPostWithComments(id) * GetRecentPostsWithCommentCount(id) in almost ANY type of data storage efficiently! - I don't get you thinking about "document" database for example: why you think that it's a problem if both GetPost(id) and GetPostWithComments(id) will return SAME? OK, they return same for document database AND ??? What problem it generate? For regular database you do Joins, for document - you just get it all together :D with embedded comments in one trip to storage! :) Never see that somebody actually complain that he get too many data in one "step" :D Sure traffic increased (and maybe little operation time ), but if we be "pragmatic" and know that even best blog posts have say max like 100 comments I don't see big issue in this :D Ah, and if you want really, really to avoid this additional "traffic" - just create 2 "document" storage's - one that store ONLY posts (without integrated comments) and another that store both and use each one in own method :) I.e. Denormalize! Regarding GetRecentPostsWithCommentCount(id) first I don't understand what it should do :D I.e. get recent posts with comments qty, but what is 'id'? Anyway, let's think it should return amount of posts with comments (i.e. say ONLY with comments) :) Agree that method here have small issues with "document" / noSQL storage's, BUT it's again possible to implement it efficiently! How you implement it in relational database? Yes, you use "count", "where" and "join" :) I.e. it's anyway "complex" (for database engine, not for developer). Now because "document" / noSQL storage's "thin" (i.e. usually lack features like Joins, Limit and Count) it's a bit more work for us developers! FOR EXAMPLE (only for example as sure algorithm is stupid): we create some key in say "hashtable1" storage, say "PostsWithCommentCount" and store here how many such posts we have that have comments :) So next, when comment created we looks up in another hashtable "PostsWithComments" id of just created post. If it's exists we don't do anything, if it does not exists, we add such key into PostsWithComments hashtable and increase value in "hasthable1" for key PostsWithCommentCount by 1 :) (sure locks etc still issue, but I skip it for now for simplicity). At the end hashtable "PostsWithComments" will contain all posts that have at least one comment and we will have simple one trip to storage into "hashtable1" to retrive value for "PostsWithCommentCount" key. So to made it work, we just use actually 2 hashtables (can be actually one sure thing) and use some stupid (really very very stupid algorithm) that give us ability to calculate what we need :) So using a) denormalization b) parallel processing c) efficient hashing and other technologies we almost ALWAYS can implement almost ANY abstract interface efficiently (sure if this interface itself correct). Even such stuff like Paging is possible! Read memcached google group to get a lot of "algorithms" how to work efficiently with "hashtables" and implement here whole a LOT of things having only "key" and "value" in your hands :D P.S. Please don't blame me on algorithm (I know it's stupid) - I think up it right when I type it after working 10 hours already and by effected by Azure Table Storage where it's really "cheap" to denormalize data and store twice more information, but really "expensive" to made twice more calls to storage engine (i.e. you pay much more for trips to ATS, then for data capacity!) :) I am sure guys here will think up 1000s times better stuff :)

evereq commented on Abstracting the persistence medium isn’t going to let you switch persistence abstractions

Tue, 03 Aug 2010 20:16:38 GMT

Nice post, nice analyze, just some horrible wrong conclusions, sorry :) First my reaction was just to not comment (maybe was really good idea :)) But then I decide - I should :) a) You seems got Joel Spolsky horrible WRONG :) He never say "don't even try" (at least I cannot find where, point me if I am wrong) If guys couple of decades ago "don't try", we don't get strings in C++, we don't get now Internet (TCP/IP), we don't have ALL mathematics (as actually it's just leaky abstractions :D) and other brilliant (and not so brilliant) things! Same like if you don't try to develop NHibernate ("mapping" abstraction actually that works well for RDBMS) or Rhino Mocks, .NET developers will not get now both, just because abstractions are leaky! Ayende, you really think that we don't need to TRY to create abstractions, because most of them (i.e. not trivial) are leaky !!!???? Correct this, PLEASE :) in post so guys near you continue to TRY or correct me if I get it wrong :) b) I will repeat: Joel Spolsky was absolutely right - abstractions are leaky, the same way like I study 10+ years ago, that all "not trivial" sets of axioms in mathematics are "leaky" ! (read [en.wikipedia.org/.../G%C3%B6del%27s_incompleten...](http://en.wikipedia.org/wiki/G%C3%B6del%27s_incompleteness_theorems) if interested). It's same like all developers probably know (at least professional) that there is no silver bullet etc.. All this "concepts" ("one full solution for all cases", "complete and not leaky abstractions" etc) are same! The problem here that your readers should know that while abstractions can be leaky - they also can be useful! So TRY to create them, IF needed, even if it's hard sometimes :)

Jonty commented on Abstracting the persistence medium isn’t going to let you switch persistence abstractions

Tue, 03 Aug 2010 09:43:06 GMT

Urgh, how do you do angle brackets in the comments? repository.Get < IPostWithComments > ()

Jonty commented on Abstracting the persistence medium isn’t going to let you switch persistence abstractions

Tue, 03 Aug 2010 09:41:35 GMT

My last post got spam filtered: I disagree we shouldn't abstract this. What about a fetching strategy? repository.Get

Mark Rendle commented on Abstracting the persistence medium isn’t going to let you switch persistence abstractions

Tue, 03 Aug 2010 08:28:21 GMT

Good post. In the Ruby world, there are various adapters for the DataMapper O(R)M which link to non-relational stores, which suggests to people that they can switch between RDBMS and NoSQL persistence by changing a setting. However, for all but the simplest storage requirements, there is such a disconnect between the two paradigms that the classes you write for one are essentially unusable and meaningless with for the other. In my own projects, I now use DataMapper or ActiveRecord for RDBMS mapping, and store-specific mappers like Mongoid for NoSQL backends. BTW, I've yet to work on a project that has not required a combination of RDBMS and NoSQL.

Tim Mahy commented on Abstracting the persistence medium isn’t going to let you switch persistence abstractions

Mon, 02 Aug 2010 16:30:02 GMT

@Dennis: "a custom made architecture is usually used, instead of using some nice infrastructure component." because it just can't be just an infrastructural solution...

Dennis commented on Abstracting the persistence medium isn’t going to let you switch persistence abstractions

Mon, 02 Aug 2010 16:20:10 GMT

Actually it would be great if someone could actually come up with a solution for this problem. It is very very common for an larger application to have these requirements for various parts of the architecture. And the problem is often that for the "smaller" parts, a custom made architecture is usually used, instead of using some nice infrastructure component.

Bogdan Marian commented on Abstracting the persistence medium isn’t going to let you switch persistence abstractions

Mon, 02 Aug 2010 11:46:43 GMT

@Ayende Man, you really hit me with this article... Still, we can abstract the way we are going to access a a persistent container (RDBMS, OODBMS, XML, etc.), as long as we do not want to switch it somewhere down the development/production line... So we could have all ORM under one framework (as Java Persistence API), all document based API under one framework and so on.

Ayende Rahien commented on Abstracting the persistence medium isn’t going to let you switch persistence abstractions

Mon, 02 Aug 2010 10:45:39 GMT

Jonty, That is something that is only useful for RDBMS, with storage like DocDB or KV stores, it is entirely unneeded. For BigTable storage, it isn't even _possible_.

Martin Aatmaa commented on Abstracting the persistence medium isn’t going to let you switch persistence abstractions

Mon, 02 Aug 2010 10:12:36 GMT

So, having accepted defeat on the idea of persistence ignorance, the inevitable question for every project will be: what is our persistence mechanism going to be? Does this spell the beginning of the end for the RDBMS?

Ayende Rahien commented on Abstracting the persistence medium isn’t going to let you switch persistence abstractions

Mon, 02 Aug 2010 09:12:47 GMT

Dennis, Yes, fixed.

Dennis commented on Abstracting the persistence medium isn’t going to let you switch persistence abstractions

Mon, 02 Aug 2010 09:09:27 GMT

I presume you mean "can't" in the last sentence?