Ayende @ Rahien

Ayende Rahien commented on You’ll pry transactions from my dead, cold, broken hands

Fri, 24 Jun 2011 22:19:57 GMT

AJ, Let us assume that you have just crashed in the middle of step 2. But, let us assume that you have more than a single server running. That means that you can't just "get list of pending tx, or applied tx", because there are other processes that are going to be actually processing it. What it goes down to is that because MongoDB doesn't have transactions, you have to build your own distributed transaction coordinator with the basic blocks of atomic swap. I am sorry, but I see no point in which it make sense to do something like that for real software. I am willing to bet that most people's attempt to write a transaction manager are going to be riddled with holes for a variety of edge cases, and that is even before we include the fast that they actually recommend adding business logic to the transaction handler part.

AJ commented on You’ll pry transactions from my dead, cold, broken hands

Fri, 24 Jun 2011 21:43:19 GMT

The documentation says 'These "repair" jobs should be run at application startup and possibly at regular interval to catch any unfinished transaction.' 1. any failure between after step 1 and before step 3: Application should get a list of transactions in state "pending" and resume from step 2. 2. any failure after step 3 and before step 5: Application should get a list of transactions in state "applied" and resume from step 4. Repair jobs will repair if something is in failed state. If not, no action is taken by that repair step. I will expand on it later.

Ayende Rahien commented on You’ll pry transactions from my dead, cold, broken hands

Fri, 24 Jun 2011 21:18:10 GMT

AJ, who is going to restart this step? Where is the transaction coordinator? Where is the information about the tx itself is stored?

AJ commented on You’ll pry transactions from my dead, cold, broken hands

Fri, 24 Jun 2011 17:14:37 GMT

Step 2 is idempotent. A failure of it midway can be simply restarted. It only pushes if not already pushed. So a repeat of step 2 will not harm (by duplication). All the steps are either atomic or idempotent. Crash is handled either through restart or rollback at each step (detailed in the documentation). Sure it is not pretty or easy. But for rare transactional need on a non-transactional database, one can give it a serious thought. I am not comparing to RavenDB (or other RDBMS) true transactional feature. Just that it was not tried out well enough in MongoDB by your client.

Ayende Rahien commented on You’ll pry transactions from my dead, cold, broken hands

Fri, 24 Jun 2011 16:55:47 GMT

AJ, You are kidding, right? This still doesn't solve the problem of crashing midway, consider the case of a failure in the middle of step 2. More to the point, this is a LOT of code, it is VERY complicated, it has *tons* of failure scenarios, hard to detect bugs, etc. Sorry, the fact that you can hop on one leg from New York to Las Vegas doesn't imply that this is a viable means of transportation.

AJ commented on You’ll pry transactions from my dead, cold, broken hands

Fri, 24 Jun 2011 13:56:49 GMT

Refer to http://www.mongodb.org/display/DOCS/two-phase+commit for 10gen's suggested way to handle multi-doc transaction with MongoDB.

Ayende Rahien commented on You’ll pry transactions from my dead, cold, broken hands

Wed, 22 Jun 2011 06:57:44 GMT

Marcel, Yes, this has been implemented for production To be fair, it is a stopgap measure while they were researching a better alternative

Marcel Konnegen commented on You’ll pry transactions from my dead, cold, broken hands

Wed, 22 Jun 2011 06:42:11 GMT

Hi Ayende, I usually don't post very often, but this time, I must say that I am absolutely shocked by how any serious software developer could produce such a "horse excrement" (as you mentioned) in a productive environment. Has the above code really been implemented in a productive environment???

Ayende Rahien commented on You’ll pry transactions from my dead, cold, broken hands

Tue, 21 Jun 2011 16:00:49 GMT

Chris, Well, yes, but while it is also possible to walk from Los Angeles to Chicago, you don't see people do that very often. In fact, by most people perception, "you can't walk from Los Angeles to Chicago" is a true statement.

Chris Wright commented on You’ll pry transactions from my dead, cold, broken hands

Tue, 21 Jun 2011 15:56:34 GMT

You can definitely write transaction support on top of a non-transactional data store. It just takes far too much time to be useful for most people whose product is not a transactional data store.

Neil commented on You’ll pry transactions from my dead, cold, broken hands

Tue, 21 Jun 2011 11:13:10 GMT

Thanks Ayende. Makes sense really, choose your db based on your essential requirements.

Ayende Rahien commented on You’ll pry transactions from my dead, cold, broken hands

Tue, 21 Jun 2011 10:50:36 GMT

Neil, I am not. You can't simulate transactions if the db doesn't support it. Then, you are left with either: - Avoid requiring transactions (which can be hard, but is possible) - Choose a db that supports them

Neil commented on You’ll pry transactions from my dead, cold, broken hands

Tue, 21 Jun 2011 10:48:38 GMT

Ayende, Understood that it's a simplification of the real problem. As you say, the original snippet is ugly and brittle and I agree that trying to approximate transactions is a not an ideal solution. How then, would you propose working around the issue if changing the entire persistence mechanism is not a justifiable option?

Ayende Rahien commented on You’ll pry transactions from my dead, cold, broken hands

Tue, 21 Jun 2011 08:18:45 GMT

Daneil, Take a look at RaccoonBlog sample app (which is also powering this blog), this is an example of what I consider to be well designed RavenDB application. The basic idea is that you don't really need to worry about session life cycle and calling save changes in the controller.

Daniel Lang commented on You’ll pry transactions from my dead, cold, broken hands

Tue, 21 Jun 2011 07:38:13 GMT

Ayende, I get the point that Raven ist probably the best document-db, but what do you mean with "This isn’t the best RavenDB code, ideally I wouldn’t have to create the session here"? I would like to know, how I could write the same code even better / shorter? Many thanks in advance!

Ayende Rahien commented on You’ll pry transactions from my dead, cold, broken hands

Tue, 21 Jun 2011 07:28:25 GMT

Neil, The example is intentionally over simplified, to make a point. Yes, there are better ways of doing that, as for "not truly transactional", that is a scary concept. Having transactions is like being pregnant, you can't be half & half.

Ayende Rahien commented on You’ll pry transactions from my dead, cold, broken hands

Tue, 21 Jun 2011 07:26:59 GMT

Demis, You are correct that there are some other NoSQL dbs out there that offer transactions, but most often, one of the laments against NoSQL is that there are no transactions

meisinger commented on You’ll pry transactions from my dead, cold, broken hands

Tue, 21 Jun 2011 03:51:03 GMT

@Neil +1 I was thinking the same exact thing... why not "event source" it and be done? And while it is not in a nice "transaction" it could still be err... transactional. It almost sounds like they were trying to get a "consistent read" out of it...

Neil commented on You’ll pry transactions from my dead, cold, broken hands

Tue, 21 Jun 2011 03:31:24 GMT

I'm not particularly well versed in either NoSQL or DDD/CQRS but would this scenario be a candidate for event sourcing? It seems as if storing a PostAdded document could offload the state management and transactional logic to some other process. If interrupted, said process could simply pick up where it left off. Not truly transactional I know, but could remove some of the issues with original code snippet.

Demis Bellot commented on You’ll pry transactions from my dead, cold, broken hands

Tue, 21 Jun 2011 01:51:18 GMT

“We tried using NoSQL, but we are moving to Relational Databases because they are easier…” I think this article should probably make mention that there are other NoSQL databases that support transactions since it's a little misleading as-is: http://nosql.mypopescu.com/post/6732339201/multi-document-transactions-in-ravendb-vs-other-nosql In the NoSQL space, there are a couple of other solutions that support transactions: - Google Megastore - Redis has two mechanisms that come close to transactions: MULTI/EXEC/DISCARD and pipelining —this one is exemplified in this Redis based triplestore database implementation - many of the graph databases (Neo4j, HyperGraphDB, InfoGrid)

SSII Lyon commented on You’ll pry transactions from my dead, cold, broken hands

Mon, 20 Jun 2011 22:29:33 GMT

<> Doesn't that simply come from the fact that 99.9% of everything taught in school and in books is on the relational model? Of course if the skillset and mindset of IT people are formatted to be relational, relational is going to seem easier. Does anyone know of a good primer on document DBs that could help alleviate this?

jimmy zimmerman commented on You’ll pry transactions from my dead, cold, broken hands

Mon, 20 Jun 2011 19:20:05 GMT

>but at this point I think that I am busy explaining why horse excrement isn’t suitable for gourmet food Greatest. Comment. Ever. +D

Ayende Rahien commented on You’ll pry transactions from my dead, cold, broken hands

Mon, 20 Jun 2011 18:22:51 GMT

Peter, That really depend, usually we are talking about document model in aggregates, but there are a lot of associations between those aggregates. In most scenarios, you probably are wrong to require mutli docs transactions, because it is okay to do this without multi doc transactions in most cases. Most of the time it is an indication of bad aggregates boundaries, but there are good reasons to want to have multiple documents (for example, different reasons for updating something in the same aggregate means that this is split to two documents) that you then need to modify in tandem. This is usually the case of practical reasons causing the splitting of a single aggregate.

Peter commented on You’ll pry transactions from my dead, cold, broken hands

Mon, 20 Jun 2011 17:54:36 GMT

Ayende, Thanks for the response. I was genuinely curious, not trying to score a point on either side. I think you're right on and in practice it is much better to support multi-doc transactions in those scenarios. From a purely theoretical modeling standpoint, do you think it's fair to say that if you are using a document database and have a lot of mult doc transactions, that's probably a warning sign?

Ayende Rahien commented on You’ll pry transactions from my dead, cold, broken hands

Mon, 20 Jun 2011 17:14:10 GMT

Peter, * RavenDB has the notion of locality, so even when you have multiple servers, you can still keep related data local, so you can get multi doc transactions. * In multiple servers, you still have DTC, it isn't recommended, yes. But it might be suitable for rare cases. Note that RavenDB allows you to grow from a single server (itself able to serve a lot of data) to multiple servers. That growth means changes to your application, certainly. But I think that it is better than saying "this feature is hard to implement using shards, we won't allow it ever"

Peter commented on You’ll pry transactions from my dead, cold, broken hands

Mon, 20 Jun 2011 17:05:56 GMT

So the general idea here is MongoDB does not support multiple document transactions, RavenDB does. However, as you mention, if you have to shard and don't/can't localize your documents, you have to use distributed transactions, which you seem to recommend against. If somebody reached that point with RavenDB, many shards, non local documents, what would you recommend? You'd have to change the model right? Or if possible just use map-reduces for counts and the like. In other words, if your data gets big enough, you'll probably run against this issue anyway, Mongo or Raven?

Ayende Rahien commented on You’ll pry transactions from my dead, cold, broken hands

Mon, 20 Jun 2011 16:35:39 GMT

Mike, Thanks, minor detail, but I fixed it.

Mike McG commented on You’ll pry transactions from my dead, cold, broken hands

Mon, 20 Jun 2011 15:05:28 GMT

You forgot to pass 'userId' into 'session.Load()'.

Ayende Rahien commented on You’ll pry transactions from my dead, cold, broken hands

Mon, 20 Jun 2011 13:24:39 GMT

Ajai, As I said, that is really an issue of how you want to deal with things. In their scenario, it absolutely made sense to have it happen in this fashion. The blog model is a very simple one, and one that is very easy to work with and explain, but it is not something that you can say: "NEVER lose this data". The actual information scenario did require them to have all or nothing semantics, please do not try to read too much into the sample scenario, it is intentionally simplified to make it easy to understand.

Ajai commented on You’ll pry transactions from my dead, cold, broken hands

Mon, 20 Jun 2011 13:19:17 GMT

Nice, that comment that I spent half hour typing out was rolled back because some stupid counts could not be updated :) I'd blame your client here, load and save User, Post to update some count?? Mongo $inc anyone? And by the way was surprised to read Mongo has a global reader writer lock across collections! Choose wisely, but guess if it's good enough for Foursquare it is good enough for rest of us.... Ajai