http://ayende.comAyende @ RahienCopyright (C) Ayende Rahien 2004 - 2021 (c) 202660Mark, There is a _huge_ difference in having a secondary index in an external data store and actually having the relevant _data_ in a separate data store. Note the terms I am using, secondary vs. separate, index vs. data. I am quite familiar with both Red Hat and JBoss, and you can safely assume that I have some knowledge about Hibernate. And while I will freely admit to being focused on the .NET stack, I am most certainly not "not looking at anything not happening in the Microsoft world". Abstraction is something that you have to approach carefully. It has its own set of costs, and you have to be aware of leaky abstractions.http://ayende.com/153699/ask-ayende-repository-for-abstracting-multiple-data-sources#comment44http://ayende.com/153699/ask-ayende-repository-for-abstracting-multiple-data-sources#comment44Thu, 19 Jan 2012 03:48:27 GMTOn the Lucene/Neo4j - Correct. Well maybe, because they might not contain all the same data. But they do work "seemlessly". I was just pointing that out so don't get caught up in that detail. Look at the bigger picture. The Neo4J/RDMBS is about multiple datastores. Think about the context and don't nit pick it so much. I was just throwing things out there. As for the second example, understand your fear. Is it right for all things? No. Does abstracting away lead to trouble? Not always. Isn't NHibernate abstracting RDBMS? Additionally the Repository and DAO patterns abstract the data store. FYI, JBoss (aka Red Hat) is the company that manages Hibernate... I not surprised you are not familiar with it. Then again, I am sort of surprised you are not. What I mean is that you are a pretty smart guy (so i would think you would keep up on things), but you seem to be in the .NET box (i.e. not looking at anything not happening in the Microsoft world).http://ayende.com/153699/ask-ayende-repository-for-abstracting-multiple-data-sources#comment43http://ayende.com/153699/ask-ayende-repository-for-abstracting-multiple-data-sources#comment43Thu, 19 Jan 2012 01:37:29 GMTMark, The Neo4j reference and the Lucene example aren't about multiple data stores, not logically. They are simply replica of the information in a way that is easier to consume in some manner. No different from creating a lookup table or denormalization in RDBMS. The first example I am not familiar with, but just from the description, it frightens me. Different data sources have different behaviors, abstracting that away leads to trouble. http://ayende.com/153699/ask-ayende-repository-for-abstracting-multiple-data-sources#comment41http://ayende.com/153699/ask-ayende-repository-for-abstracting-multiple-data-sources#comment41Thu, 19 Jan 2012 00:59:33 GMTSome examples of "mixing" data sources: http://www.jboss.org/teiid - "Teiid is a data virtualization system that allows applications to use data from multiple, heterogenous data stores" On this next link, Look at "Chapter 22. Cross-store persistence" for storing part of the data in a Graph Database and the rest in a RDBMS http://static.springsource.org/spring-data/data-graph/snapshot-site/reference/pdf/spring-data-neo4j-reference.pdf - Additionally, you can store your data in an RDMBS (or non-rdbms) and use an Index like Lucene as a datastore. In fact, the above example does all 3. Another example would be to store some things in an Content repository (for you Java people - JCR) with metadata and the rest in RDMBS (or pick your datastore). Also take a look at http://www.jboss.org/modeshape http://ayende.com/153699/ask-ayende-repository-for-abstracting-multiple-data-sources#comment40http://ayende.com/153699/ask-ayende-repository-for-abstracting-multiple-data-sources#comment40Thu, 19 Jan 2012 00:52:50 GMTFrank, Bounded contexts shift over time, but very very slowly, and usually as a result of shifting responsibilities inside the org itself. Bounded contexts represent how the org view itself, a reflection of how it behaves. If you have a system whose contexts are drastically different than the internal organization structure, it is a bad system for the org, and likely to be causing problems. You are giving the wrong example with the iPad, that isn't a single system, it is an OS with a shell, no commonality except UI guidelines. When talking about bounded contexts, we always talk about them inside a system / application, not generically.http://ayende.com/153699/ask-ayende-repository-for-abstracting-multiple-data-sources#comment39http://ayende.com/153699/ask-ayende-repository-for-abstracting-multiple-data-sources#comment39Wed, 18 Jan 2012 18:09:47 GMTRiccardo, Again, the problem is in the problem statement. I don't believe in UI and DB developers. The very fact that you structured your team this way means that you already structured your project this way. See Conway's Law. You shouldn't do things like that up front, first choose an architecture, then build the team structure. And I would strongly recommend on vertical division, not horizontal division.http://ayende.com/153699/ask-ayende-repository-for-abstracting-multiple-data-sources#comment38http://ayende.com/153699/ask-ayende-repository-for-abstracting-multiple-data-sources#comment38Wed, 18 Jan 2012 18:06:56 GMT@JamesMcKay I fully believe in YAGNI. But also know that sometimes a problem changes and you do need it. Case in point: my current system uses an ORM because of a mandated change from Oracle to SQL. Previously, it was SP based due to some misguided architecture from the original devs 8 years ago. This system was migrated without a big bang approach. At the point of SQL GoLive, the system could run on either database.http://ayende.com/153699/ask-ayende-repository-for-abstracting-multiple-data-sources#comment37http://ayende.com/153699/ask-ayende-repository-for-abstracting-multiple-data-sources#comment37Wed, 18 Jan 2012 12:24:27 GMTAyende, I didn't want to insinuate that accessing a DWH in a OLTP context is a good idea. What I am trying to say is that your bounded contexts may shift for various forces of change. Short of doing great migration projects a (maybe interim) solution may be to have UIs accessing several systems. With regard to the modular approach of UIs - I think there is some handwaving done at that end of the system. If you look at an iPad, that's certainly a modular UI with pretty weak integration between the different modules. If you imagine some process spanning several bounded contexts, the UX experience would be rather disgusting.http://ayende.com/153699/ask-ayende-repository-for-abstracting-multiple-data-sources#comment36http://ayende.com/153699/ask-ayende-repository-for-abstracting-multiple-data-sources#comment36Wed, 18 Jan 2012 10:30:24 GMTIf I'm a developer responsible of the database, I'm not happy if UI developers can interact freely with a model exposed from a context. I'd like that UI developers are allowed to interact with a fixed and easy to control interface as a repository is.http://ayende.com/153699/ask-ayende-repository-for-abstracting-multiple-data-sources#comment35http://ayende.com/153699/ask-ayende-repository-for-abstracting-multiple-data-sources#comment35Wed, 18 Jan 2012 10:05:01 GMTChris - that doesn't make much sense. There are dozens of plugins you can embed in your site to get chat, ecommerce, and blogging. This has no relation to choosing to use the repository pattern. It's purely a UI concern. Sure you might need integrated authentication, but your main app should have no idea that these other services exist. They are aspects of the web experience.http://ayende.com/153699/ask-ayende-repository-for-abstracting-multiple-data-sources#comment34http://ayende.com/153699/ask-ayende-repository-for-abstracting-multiple-data-sources#comment34Wed, 18 Jan 2012 00:26:13 GMTon a related note, when is macto going to see the day light? I'm really anxious to see the code.http://ayende.com/153699/ask-ayende-repository-for-abstracting-multiple-data-sources#comment33http://ayende.com/153699/ask-ayende-repository-for-abstracting-multiple-data-sources#comment33Tue, 17 Jan 2012 22:05:28 GMTNot sure that I understand you clearly. I'm building this into the app, because these are the requirements. But my guess it that we have a bit different definitions of 'building into'. For me it means 'providing the functionality' User definitions are the same here (yes I know that this is a strong assumption, but I had the comfort of creating this from scratch so no collision here) Modular approach - I totally agree, but again, I have a feeling that we have different things in mind. Can you elaborate on that?http://ayende.com/153699/ask-ayende-repository-for-abstracting-multiple-data-sources#comment32http://ayende.com/153699/ask-ayende-repository-for-abstracting-multiple-data-sources#comment32Tue, 17 Jan 2012 21:55:09 GMTChris, Huh? Why are you building this into the app? Those are separate apps that needs to integrate at the UI level. A module based approach would be much better. Not to mention that something as simple as different user definitions would require you to do a LOT of work even if you could try to abstract things awayhttp://ayende.com/153699/ask-ayende-repository-for-abstracting-multiple-data-sources#comment31http://ayende.com/153699/ask-ayende-repository-for-abstracting-multiple-data-sources#comment31Tue, 17 Jan 2012 21:08:32 GMTSure - embedding blogging, e-comerce or live chat capabilities into a custom web app. This is actually something that I had to do recently.http://ayende.com/153699/ask-ayende-repository-for-abstracting-multiple-data-sources#comment30http://ayende.com/153699/ask-ayende-repository-for-abstracting-multiple-data-sources#comment30Tue, 17 Jan 2012 18:28:44 GMTSorry, my last phrase should be intended in the opposite way. This is what i intended: "Last thing, just to be clear: it probably hurts more to *use* a repo when you should *not*, then to *not* use it when you actually *may* have use it." Sorry for the rapid triplette.http://ayende.com/153699/ask-ayende-repository-for-abstracting-multiple-data-sources#comment29http://ayende.com/153699/ask-ayende-repository-for-abstracting-multiple-data-sources#comment29Tue, 17 Jan 2012 18:25:14 GMTI'll add: you originally quoted the phrase "... if you have several kind of data sources in different technologies ..." and that is *exactly* my situation 1 year ago. I've used that thin repo abstraction, it worked wonderfully, and everything went fine. And there were *no* better alternative that i can speak of. So, i give to you guys my own experience with that, saying that *sometimes* it may be useful. Last thing, just to be clear: it probably hurts more to *not* use a repo when you actually *can*, then to *use* it when you should *not*.http://ayende.com/153699/ask-ayende-repository-for-abstracting-multiple-data-sources#comment28http://ayende.com/153699/ask-ayende-repository-for-abstracting-multiple-data-sources#comment28Tue, 17 Jan 2012 18:21:38 GMTIn general, i tend not to use words like "always" or "never", unless accompanied by "except in rare cases". Because, i mean, people may think to actually NEVER or ALWAYS do this or that things. Saying NEVER use repos, where instead there are cases where it may makes sense, is missing something. I know it may sounds like nitpicking, it's just that it is happened to me a lot of times to hear people say stuff like "in university, they teach us to NEVER denormalize data!" or "he said to NEVER use a nosql" or, like in this case, repos. Just that, cheers and keep up the good work with Raven & co.http://ayende.com/153699/ask-ayende-repository-for-abstracting-multiple-data-sources#comment27http://ayende.com/153699/ask-ayende-repository-for-abstracting-multiple-data-sources#comment27Tue, 17 Jan 2012 18:18:14 GMTChris, Can you really think about common scenarios where you actually reuse a controller? Controllers are really part of the application, they are about what the application is doing. Moving them to another app makes very little sense. http://ayende.com/153699/ask-ayende-repository-for-abstracting-multiple-data-sources#comment26http://ayende.com/153699/ask-ayende-repository-for-abstracting-multiple-data-sources#comment26Tue, 17 Jan 2012 18:10:44 GMTLet's take the example I had earlier - so a controller with capabilities for doing some operations on objects of a single type. If you have a dependency in it for a session or a db context then it's not possible to reuse that controller elsewhere (e.g in a different application). If on the other hand, the controller would have a dependency on a repository interface, then it would be reusable, provided that the session (or whatever is used to wrap the session) implements that particular interface. So my question is how would you achieve this type of re-usability?http://ayende.com/153699/ask-ayende-repository-for-abstracting-multiple-data-sources#comment25http://ayende.com/153699/ask-ayende-repository-for-abstracting-multiple-data-sources#comment25Tue, 17 Jan 2012 18:06:24 GMTNjy, I do consider those rare cases, but unless I actually have a reason to speak about those scenarios, why do so? Again, "never hitchhike". http://ayende.com/153699/ask-ayende-repository-for-abstracting-multiple-data-sources#comment24http://ayende.com/153699/ask-ayende-repository-for-abstracting-multiple-data-sources#comment24Tue, 17 Jan 2012 18:04:31 GMTAgain, yes and no, imho. > Trying to say, "you do it only if" will cause people > to turn their head off for the if and just go ahead > and do the thing that you don't want. Yep Ayende, stupid people are everywhere :-) ! I mean, the same could be said for "don't use RDBMS if..." or "don't use RavenDB if...". What i'm saying is that you are actually quite right but, since you seem to not consider the possibility of a case where repos may make makes sense, i just pointed that out (that is, in fact, there are cases where it could make sense). Case in point: you seem to take for granted that you *can* model the data storage and it's infrastructure: in one of my previous projects (last year) i had to work with external data, provided by third parties, and available *only* as a custom xml feed from a set of remote jsp webservices. Being able to let the team work with that data in a standard and consolidated way (even before the data was available, using fake in-memory data) was a huge win for us. In the end, i repeat: you are correct for almost all of the *common* cases, but consider other casistics and situations, they happen too. Whaddaya think?http://ayende.com/153699/ask-ayende-repository-for-abstracting-multiple-data-sources#comment23http://ayende.com/153699/ask-ayende-repository-for-abstracting-multiple-data-sources#comment23Tue, 17 Jan 2012 18:00:50 GMTChris, I don't understand the question, can you expand on that?http://ayende.com/153699/ask-ayende-repository-for-abstracting-multiple-data-sources#comment22http://ayende.com/153699/ask-ayende-repository-for-abstracting-multiple-data-sources#comment22Tue, 17 Jan 2012 17:53:05 GMT@Ayende, fair point. Let me rephrase that - do you consider the fact of passing 'to much' along with the session a problem? If so, how would you fix that?http://ayende.com/153699/ask-ayende-repository-for-abstracting-multiple-data-sources#comment21http://ayende.com/153699/ask-ayende-repository-for-abstracting-multiple-data-sources#comment21Tue, 17 Jan 2012 17:51:39 GMTNjy, You make a wrong assumption here, you are starting out by assuming that the rare scenario is in effect here. I am assuming that this is the common scenario, and only if I get additional data, then I'll have answers for the rare scenario. Trying to say, "you do it only if" will cause people to turn their head off for the if and just go ahead and do the thing that you don't want. In the same sense that you say "never hitch a ride with a stranger", emphasis on the never. But there are a lot of exceptions for that. For example, if you are in a burning field and the only way out is to hitch a ride on a car driven by someone you don't know or be burned alive... But that isn't what you focus on.http://ayende.com/153699/ask-ayende-repository-for-abstracting-multiple-data-sources#comment20http://ayende.com/153699/ask-ayende-repository-for-abstracting-multiple-data-sources#comment20Tue, 17 Jan 2012 17:48:17 GMTChris, The law of demeter is nice, except that it ignores a lot of other factors. For example, with the RavenDB session, we moved a lot of operations in the session.Advanced property, just because it made the intellisense easier. Fluent interfaces are also in viloation of demeter, etc, etc. http://ayende.com/153699/ask-ayende-repository-for-abstracting-multiple-data-sources#comment19http://ayende.com/153699/ask-ayende-repository-for-abstracting-multiple-data-sources#comment19Tue, 17 Jan 2012 17:45:29 GMTAlireza, That is because I am not doing that. I am testing my code against an in memory databasehttp://ayende.com/153699/ask-ayende-repository-for-abstracting-multiple-data-sources#comment18http://ayende.com/153699/ask-ayende-repository-for-abstracting-multiple-data-sources#comment18Tue, 17 Jan 2012 17:43:04 GMTSatish, Depending on what you are doing, yes. While I wouldn't categorically say it is wrong, I dislike it. That is, Shared Database is usually going to lead you into problems down the road. Your database becomes your contract, and you can't make changes there, you have to coordinate with many people on anything that happens, you have to worry about other people messing you up, etc. I much rather have isolated databases and a shared reporting DB.http://ayende.com/153699/ask-ayende-repository-for-abstracting-multiple-data-sources#comment17http://ayende.com/153699/ask-ayende-repository-for-abstracting-multiple-data-sources#comment17Tue, 17 Jan 2012 17:42:30 GMTFrank, A data warehouse is typically a good thing to do for reporting concerns, yes. For OLTP / operational ones? Not so much. Trying to have a unified data model is almost always a mistake, for versioning, concurrency, management and coordination reasons.http://ayende.com/153699/ask-ayende-repository-for-abstracting-multiple-data-sources#comment16http://ayende.com/153699/ask-ayende-repository-for-abstracting-multiple-data-sources#comment16Tue, 17 Jan 2012 17:41:03 GMTYes. And no. Your advice that it should paying more attention to the question itself is good. Your advice that *usually* it is not the right track, even if it seems so, it's good too. The fact that you seem not to consider a case where this may be applicable is, well, simply wrong. There's a value in being able to unify the data acces strategy among heterogeneous datasources (if used carefully, and i repeat: IF USED CAREFULLY).http://ayende.com/153699/ask-ayende-repository-for-abstracting-multiple-data-sources#comment15http://ayende.com/153699/ask-ayende-repository-for-abstracting-multiple-data-sources#comment15Tue, 17 Jan 2012 16:55:48 GMT@Ayende, I agree on the fact that dropping the repository in favor of using a pure O/RM simplifies the app, but on the other hand, doesn't it violate the law of demeter? My example would be an asp.net mvc controller for handling operations on a Foo object with the pure session/context passed in - allows you to do what you want, but why should a Foo controller have access to a Bar collection which is exposed in the session/context (assuming that it doesn't want to do anything with it)?http://ayende.com/153699/ask-ayende-repository-for-abstracting-multiple-data-sources#comment14http://ayende.com/153699/ask-ayende-repository-for-abstracting-multiple-data-sources#comment14Tue, 17 Jan 2012 16:43:15 GMT