http://ayende.comAyende @ RahienCopyright (C) Ayende Rahien 2004 - 2021 (c) 202660Cool. Created the issue: http://issues.hibernatingrhinos.com/issue/RavenDB-969http://ayende.com/161218/rob-s-sprint-idly-indexing#comment21http://ayende.com/161218/rob-s-sprint-idly-indexing#comment21Fri, 01 Mar 2013 14:27:21 GMTDamian, We have 2K+ tests, most of them with some form of indexes. We run them a LOT. any saving there would be useful in general.http://ayende.com/161218/rob-s-sprint-idly-indexing#comment20http://ayende.com/161218/rob-s-sprint-idly-indexing#comment20Fri, 01 Mar 2013 13:05:47 GMTActually, that may be a nice-to-have from a production pov. An index that is deleted and then re-created, assuming it is exactly the same, would be slightly faster. Don't know how often that would happen though really.http://ayende.com/161218/rob-s-sprint-idly-indexing#comment19http://ayende.com/161218/rob-s-sprint-idly-indexing#comment19Fri, 01 Mar 2013 12:41:46 GMTYeah, that sounds good too. Generate a hash from the source, use it as the CompilerParameters.OutputAssemblyName and if the assembly already exists on disk (in a location that will exist between test sessions i.e. users temp dir) load it. Or something like that :)http://ayende.com/161218/rob-s-sprint-idly-indexing#comment18http://ayende.com/161218/rob-s-sprint-idly-indexing#comment18Fri, 01 Mar 2013 12:37:59 GMTDamian, In that case, how about implementing on disk caching for this?http://ayende.com/161218/rob-s-sprint-idly-indexing#comment17http://ayende.com/161218/rob-s-sprint-idly-indexing#comment17Fri, 01 Mar 2013 12:24:04 GMTYes, the cost per test run, where I am run _one test at a time_, in the usual TDD(-ish) scenario. The index compilation caching (which is great) only kicks in when I run 2 or more tests per session. http://i.imgur.com/38DF0fc.png - second test benefits from the caching. My other approach is to be able to supply a predicate to my application so the test fixture can configure it to only create indexes that are going to be used. But that means my acceptance test fixtures need to know what indexes may be required which I find to be leaky. (I take a different approach with my unit tests, no problems there) Yes, it's a development pain and not a production issue. I may be an edge case though.http://ayende.com/161218/rob-s-sprint-idly-indexing#comment16http://ayende.com/161218/rob-s-sprint-idly-indexing#comment16Fri, 01 Mar 2013 12:16:32 GMTOh, you are talking about the cost _per test run_, right? I was thinking about production runs, actually. In that case, can't you handle this via the index compilation caching that we already have?http://ayende.com/161218/rob-s-sprint-idly-indexing#comment15http://ayende.com/161218/rob-s-sprint-idly-indexing#comment15Fri, 01 Mar 2013 11:52:35 GMTDamian, There is really no cost in doing the compilation (it happens once, and that is it.)http://ayende.com/161218/rob-s-sprint-idly-indexing#comment14http://ayende.com/161218/rob-s-sprint-idly-indexing#comment14Fri, 01 Mar 2013 11:51:40 GMTRavenDB already caches compiled indexes ( https://github.com/ayende/ravendb/blob/master/Raven.Database/Linq/QueryParsingUtils.cs#L334 , discussion https://groups.google.com/d/msg/ravendb/hsMc4lLnaXU/h0WRLOYog9EJ ) which makes second and subsequent test runs that use create that index much faster. I'm wondering if it would be possible to configure the indexes to be lazily compiled? That is, compiled and loaded when first queried? Am currently doing system acceptance tests where we have an increasing number of indexes and am experiencing some time pain (20-30s +) on single test runs.http://ayende.com/161218/rob-s-sprint-idly-indexing#comment13http://ayende.com/161218/rob-s-sprint-idly-indexing#comment13Fri, 01 Mar 2013 11:48:29 GMTAlex, In other words, anything that you can do through the UI can be done in code, and pretty easily, at that.http://ayende.com/161218/rob-s-sprint-idly-indexing#comment12http://ayende.com/161218/rob-s-sprint-idly-indexing#comment12Fri, 01 Mar 2013 07:18:38 GMTAlex, ALL of RavenDB functionality is exposed via REST interface, and you can do absolutely everything the studio does. After all, the studio just uses HTTP to talk to RavenDB himself, it is not a privileged client.http://ayende.com/161218/rob-s-sprint-idly-indexing#comment11http://ayende.com/161218/rob-s-sprint-idly-indexing#comment11Fri, 01 Mar 2013 07:18:23 GMTIn my still limited experience with Raven, specifically trying to work with bundles like replication and versioning. I have noticed that its not very straightforward to accomplish certain functionality without using the studio. This specific feature is not that big of a deal to us, but we would really love to see functionality like this be configurable without going through the UI. http://ayende.com/161218/rob-s-sprint-idly-indexing#comment10http://ayende.com/161218/rob-s-sprint-idly-indexing#comment10Fri, 01 Mar 2013 01:18:44 GMTAlex, No, you can't do that at creation, but you can do that immediately after.http://ayende.com/161218/rob-s-sprint-idly-indexing#comment9http://ayende.com/161218/rob-s-sprint-idly-indexing#comment9Fri, 01 Mar 2013 00:07:05 GMTConfigurator, And the reason we wait for 10 minutes on inactivity is that we don't want to get into: "we have 1 second of rest, let us start indexing all the idle indexes, which can be VERY expensive". http://ayende.com/161218/rob-s-sprint-idly-indexing#comment8http://ayende.com/161218/rob-s-sprint-idly-indexing#comment8Fri, 01 Mar 2013 00:06:25 GMTConfigurator, You can force an index to not go into idle / abandoned mode. But in general, if you have an index that is queried weekly, you can afford to wake it up and then wait for it to catch up.http://ayende.com/161218/rob-s-sprint-idly-indexing#comment7http://ayende.com/161218/rob-s-sprint-idly-indexing#comment7Fri, 01 Mar 2013 00:05:33 GMTPatrik, This is already available at: http://hibernatingrhinos.com/builds/ravendb-unstable-v2.5http://ayende.com/161218/rob-s-sprint-idly-indexing#comment6http://ayende.com/161218/rob-s-sprint-idly-indexing#comment6Fri, 01 Mar 2013 00:04:23 GMTCan we get a way to set these flags on the index creators as well? http://ayende.com/161218/rob-s-sprint-idly-indexing#comment5http://ayende.com/161218/rob-s-sprint-idly-indexing#comment5Thu, 28 Feb 2013 23:33:58 GMTChris - while not covered explicitly in the entry above, there is a flag to "force idle" and this will be exposed in the studiohttp://ayende.com/161218/rob-s-sprint-idly-indexing#comment4http://ayende.com/161218/rob-s-sprint-idly-indexing#comment4Thu, 28 Feb 2013 15:42:42 GMT"An index that was automatically set to idle will be set to normal on its first query." What if you want the index to always be an idle index? Like a reporting index that pulls tons of things together, or a crazy reporting map/reduce that is not relevant to OLTP functionality at all? http://ayende.com/161218/rob-s-sprint-idly-indexing#comment3http://ayende.com/161218/rob-s-sprint-idly-indexing#comment3Thu, 28 Feb 2013 14:45:39 GMT"an abandoned index is one that hasn’t been queried in 72 hours" - so a weekly report will never be up to date? Also, why do idle indexes wait for 10 minutes of inactivity instead of just working only when all other indexes are up to date?http://ayende.com/161218/rob-s-sprint-idly-indexing#comment2http://ayende.com/161218/rob-s-sprint-idly-indexing#comment2Thu, 28 Feb 2013 14:32:29 GMTCool, When will you push it into the unstable branch so we can test it out? http://ayende.com/161218/rob-s-sprint-idly-indexing#comment1http://ayende.com/161218/rob-s-sprint-idly-indexing#comment1Thu, 28 Feb 2013 12:12:27 GMT