http://ayende.comAyende @ RahienCopyright (C) Ayende Rahien 2004 - 2021 (c) 202660Great, thanks. I'm about to get involved in a project in it's infancy that is currently being built on Raven so I'm trying to familiarize myself with it more. I didn't like the idea of having to leave these converters all over the place every time the object model changed, so just wanted to make sure there was a way to phase them out.http://ayende.com/66563/ravendb-migrations-rolling-updates#comment33http://ayende.com/66563/ravendb-migrations-rolling-updates#comment33Wed, 09 Nov 2011 23:39:56 GMTRyan, Yes, at that point, you'll probably run a script that would convert everything. The alternative is to just keep the converter in place for all time, which is also an optionhttp://ayende.com/66563/ravendb-migrations-rolling-updates#comment32http://ayende.com/66563/ravendb-migrations-rolling-updates#comment32Wed, 09 Nov 2011 20:17:32 GMTNot sure if you are still checking these comments, but I had a ? Say you need to do a rolling update, you write the DocumentConversionListener above, and your objects start converting as you encounter them. So this is great for commonly accessed objects, but what about old/archived stuff? Say you have 1,000 Customers and 800 of them get updated in a week but 200 of them are fairly inactive. You don't want to turn off that converter until they are, but you don't want to shutdown the app just for some old data conversions either. Do you recommend just running a script against the DB to manually convert the data and then pulling out the converter? Or is there another way.http://ayende.com/66563/ravendb-migrations-rolling-updates#comment31http://ayende.com/66563/ravendb-migrations-rolling-updates#comment31Wed, 09 Nov 2011 16:52:29 GMTMike, I don't understand the questionhttp://ayende.com/66563/ravendb-migrations-rolling-updates#comment30http://ayende.com/66563/ravendb-migrations-rolling-updates#comment30Fri, 09 Sep 2011 10:50:22 GMTVery neat, but if you don't need a listener, is the schema version recorded anyway? http://ayende.com/66563/ravendb-migrations-rolling-updates#comment29http://ayende.com/66563/ravendb-migrations-rolling-updates#comment29Fri, 09 Sep 2011 09:30:23 GMTThanks :) Just roll your own here sounds easy enough.http://ayende.com/66563/ravendb-migrations-rolling-updates#comment28http://ayende.com/66563/ravendb-migrations-rolling-updates#comment28Sun, 28 Aug 2011 19:15:30 GMTDmytrii, https://github.com/ayende/RaccoonBlog/tree/master/src/RaccoonBlog.Migrationshttp://ayende.com/66563/ravendb-migrations-rolling-updates#comment27http://ayende.com/66563/ravendb-migrations-rolling-updates#comment27Sun, 28 Aug 2011 19:10:13 GMTThat makes sense. But I was curious to see how you would do that (split radical changes into smaller ones?). It would be amazing to see a write-up about doing this kind of stuff with RavenDB (analogy of Migrator.Net and similar).http://ayende.com/66563/ravendb-migrations-rolling-updates#comment26http://ayende.com/66563/ravendb-migrations-rolling-updates#comment26Sun, 28 Aug 2011 19:07:13 GMTDmytrii, And you would do pretty much the same thing in RavenDB. But that isn't the scope of this post, it is about rolling update, not point in time update. For those sort of updates, you don't do radical changes, you make things change slowly, across deployments.http://ayende.com/66563/ravendb-migrations-rolling-updates#comment25http://ayende.com/66563/ravendb-migrations-rolling-updates#comment25Sun, 28 Aug 2011 18:56:41 GMTI don't argue that it is a radical change. I wonder how it would be handled with RavenDB. For example, with SQL database I would create a migration (using Migrator.net or similar) that would change the schema accordingly and them migrate the data. This process would be automated and easily repeatable.http://ayende.com/66563/ravendb-migrations-rolling-updates#comment24http://ayende.com/66563/ravendb-migrations-rolling-updates#comment24Sun, 28 Aug 2011 18:52:58 GMTDmytrii, If you need to do that, then you don't do radical changes.http://ayende.com/66563/ravendb-migrations-rolling-updates#comment23http://ayende.com/66563/ravendb-migrations-rolling-updates#comment23Sun, 28 Aug 2011 18:48:59 GMTOren, With the radical changes, when and how would you run the migration? Doing it as one-time process is not good enough. I need to be able to run such kind of migration in multiple environments. Cheers.http://ayende.com/66563/ravendb-migrations-rolling-updates#comment22http://ayende.com/66563/ravendb-migrations-rolling-updates#comment22Sun, 28 Aug 2011 17:01:17 GMTAyende, you are right.http://ayende.com/66563/ravendb-migrations-rolling-updates#comment21http://ayende.com/66563/ravendb-migrations-rolling-updates#comment21Sun, 28 Aug 2011 09:16:32 GMTAlex, No, it would resolve that automaticallyhttp://ayende.com/66563/ravendb-migrations-rolling-updates#comment20http://ayende.com/66563/ravendb-migrations-rolling-updates#comment20Sun, 28 Aug 2011 05:54:07 GMTTobi, Have a MigrationStoreListener that would forward the call based on the type of the entity.http://ayende.com/66563/ravendb-migrations-rolling-updates#comment19http://ayende.com/66563/ravendb-migrations-rolling-updates#comment19Sun, 28 Aug 2011 05:53:29 GMT1) Rollback? Just the same way as the forward motion, just in reverse. Do the exact same thing, but reverse the steps. 2) You usually do those sort of things for one version back, which mean that at the next release, you can do the big "check & modify" for the entire db, so you don't have to deal with the 3 versions back version. 2.1) Or you can just keep all of those around and make the checks when you need them in order, based on the version of the entity.http://ayende.com/66563/ravendb-migrations-rolling-updates#comment18http://ayende.com/66563/ravendb-migrations-rolling-updates#comment18Sun, 28 Aug 2011 05:52:53 GMTDmytrii, That requires creating a separate document, probably by just replacing that with the id of the new branch. Although, I would probably do stuff like that as a one time process, since this is a pretty radical change, and not something that you can usually slip in as a gradual transformationhttp://ayende.com/66563/ravendb-migrations-rolling-updates#comment17http://ayende.com/66563/ravendb-migrations-rolling-updates#comment17Sun, 28 Aug 2011 05:50:27 GMTSteve, It's just the default behavior of the Json serializer - it tries to do its best ignoring schema differences and supplying default valuse for missing properties. I wouldn't call it self healing or self cleaning because as you have said, sometimes it's just self destructing. It would be much better to have some schema validation mechanism that is an integral part of the database and that can't be easily bypassed (by not setting up the client correctly). As an example, please have a look at how it has been solved in Persevere: http://www.sitepen.com/blog/2008/11/17/evolving-schemas-with-persevere/ http://ayende.com/66563/ravendb-migrations-rolling-updates#comment16http://ayende.com/66563/ravendb-migrations-rolling-updates#comment16Sat, 27 Aug 2011 09:07:19 GMTDo I need a listener if I move the Customer class to a different namespace?http://ayende.com/66563/ravendb-migrations-rolling-updates#comment15http://ayende.com/66563/ravendb-migrations-rolling-updates#comment15Sat, 27 Aug 2011 09:04:21 GMTHmm, the automatic nature of self healing and self cleaning could be troublesome behaviour in cases where developers didn't know better. Granted that in a perfect world, everyone should be fully versed in the capabilities of their tools, and paying due dilligence to their changes, plus testing those changes thoroughly. But if someone scoping out changes to large document stores with changes across dozens of documents, under pressure, the tool isn't helping catch situations they may have missed, it's actively hiding them. Case in point, if you ran that scenario through without the listener, the self cleaning would erase "email" and "Name", and add the new fields with default values, would it not? My point is that the tool doesn't know whether you want to discard or translate old data, and it seems rather dangerous to have it pick a behaviour arbitrarily. My personal preference would be for a tool to detect such changes and require deliberate rules for the specific change. (discard, or translate.) http://ayende.com/66563/ravendb-migrations-rolling-updates#comment14http://ayende.com/66563/ravendb-migrations-rolling-updates#comment14Sat, 27 Aug 2011 00:42:13 GMTOnce you have many different types, all handlers need to be called. They can never be removed because some entities might still not be upgraded. This could become a perf problem (calling 100 listeners for every loaded object). I would have the listeners be registered for a specific type (and optionally for all types). http://ayende.com/66563/ravendb-migrations-rolling-updates#comment13http://ayende.com/66563/ravendb-migrations-rolling-updates#comment13Fri, 26 Aug 2011 18:44:49 GMTI don't use ravendb - these are just general data migration questions 1. Is the support for rollback explicitly missing, i.e.: it is assumed that if I have made a mistake, then I code my way forward and do a new release? Assuming, that new data was created between the release and the discovery of the need for rollback to the last stable version, and I want to convert that data back to the old format (for which I have the code, written at the time I wrote this forward conversion). I want to rollback fast and reliably - no from-dusk-to-down caffeine powered coding sessions. What can I do? 2. How is the multiple changes scenario supported - there are sites I only use once every 6 months, but I'm sure they have multiple releases during that time. So when I log in, my records might be at version 1, while the current version could be 19. Is it one class per version, and each checks which version number it belongs to? Based on the snippet above, there is no framework support for that, or is there? Or one would just schedule a batch run outside the peak hours, forcing the not yet updated records to be loaded (and thus updated)?http://ayende.com/66563/ravendb-migrations-rolling-updates#comment12http://ayende.com/66563/ravendb-migrations-rolling-updates#comment12Fri, 26 Aug 2011 15:35:40 GMTSure. Let's say we have a Company with Address, company number etc. We change the model so that the compamy no longer has address. Insteaad the address is stored in a separet document - Branch. And then how would you merge Brancge back into Company. Do you see what I mean?http://ayende.com/66563/ravendb-migrations-rolling-updates#comment11http://ayende.com/66563/ravendb-migrations-rolling-updates#comment11Fri, 26 Aug 2011 13:49:22 GMTDmytrii, I don't understand the question, can you give an example?http://ayende.com/66563/ravendb-migrations-rolling-updates#comment10http://ayende.com/66563/ravendb-migrations-rolling-updates#comment10Fri, 26 Aug 2011 13:41:55 GMTY, That is why RavenDB has support for set operationshttp://ayende.com/66563/ravendb-migrations-rolling-updates#comment9http://ayende.com/66563/ravendb-migrations-rolling-updates#comment9Fri, 26 Aug 2011 13:41:15 GMTHow would you handle the situation when a property moved to two new documents and the other way around?http://ayende.com/66563/ravendb-migrations-rolling-updates#comment8http://ayende.com/66563/ravendb-migrations-rolling-updates#comment8Fri, 26 Aug 2011 13:22:42 GMTSeems pretty straightforward for "trash" fields. But how about indices on top of changed fields? I guess if you need to i.e. search by that field, you'd want you database to migrate all documents to latest format version. What if RavenDB bundled a tool that would let you register the same converters in RavenDB and let it chew documents in the background? :)http://ayende.com/66563/ravendb-migrations-rolling-updates#comment7http://ayende.com/66563/ravendb-migrations-rolling-updates#comment7Fri, 26 Aug 2011 11:26:59 GMTI didn't mean to nit-pick and I understand the scope of the code above. My point is that given enough data rolling updates can be a nightmare and dangerous. But yes, RavenDB tackles it in a very elegant way.http://ayende.com/66563/ravendb-migrations-rolling-updates#comment6http://ayende.com/66563/ravendb-migrations-rolling-updates#comment6Fri, 26 Aug 2011 10:56:30 GMTJose, Maybe, this is code that is specific for a single use case, and as such, can make a lot of assumptions.http://ayende.com/66563/ravendb-migrations-rolling-updates#comment5http://ayende.com/66563/ravendb-migrations-rolling-updates#comment5Fri, 26 Aug 2011 10:36:41 GMTWouldn't DocumentToEntity break on a Name that only has one word? Or would we get the same word on both FirstName and LastName?http://ayende.com/66563/ravendb-migrations-rolling-updates#comment4http://ayende.com/66563/ravendb-migrations-rolling-updates#comment4Fri, 26 Aug 2011 10:32:53 GMT