http://ayende.comAyende @ RahienCopyright (C) Ayende Rahien 2004 - 2021 (c) 202660Could you test a large set of data and then expect on the ratios? -d http://ayende.com/3678/how-do-you-test-a-system-that-self-optimize-itself#comment8http://ayende.com/3678/how-do-you-test-a-system-that-self-optimize-itself#comment8Fri, 07 Nov 2008 14:21:28 GMTUse Dependency Injection to supply the randomizer and ensure that the rest of the logic is deterinistic given a certain input set from the randomizer. Then you can mock the randomizer and ensure that the algorithm works correctly. I'm guessing that you'd use a standard random number generation function provided by an API supplier and in many ways you'd have to take it as given that they've achieved their stated randomness. Like most comments imply, that will test that the impleentation of the algorithm is as you had planned. Going beyond that to get a heuristic sense of whether the algorithm itself meets the design goals is a bit more tricky. Maybe this is what you were getting at? http://ayende.com/3678/how-do-you-test-a-system-that-self-optimize-itself#comment7http://ayende.com/3678/how-do-you-test-a-system-that-self-optimize-itself#comment7Thu, 06 Nov 2008 22:21:26 GMTFrom your description, it sounds like you have combination of randomness and deterministic logic. Making randomness deterministic is not particularly hard: [blogs.msdn.com/.../TestingAgainstRandomness.aspx](http://blogs.msdn.com/ploeh/archive/2007/05/11/TestingAgainstRandomness.aspx) Having done that, you should be able to test the rest of the algorithm in a deterministic manner. Since you are such a smart guy, you would obviously have thought of this yourself, so I probably misunderstood the problem... http://ayende.com/3678/how-do-you-test-a-system-that-self-optimize-itself#comment6http://ayende.com/3678/how-do-you-test-a-system-that-self-optimize-itself#comment6Thu, 06 Nov 2008 19:28:28 GMTAssuming you are talking about an Automated Trading System (ATS) where you find these things quite often. The problem with self optimizing with these systems is that they are optimizing to closely to match exactly the test data you are providing. So because of this often an out of sample data range is used to determine if the optimization is not becoming to optimized towards the test sample. Now I still find this to optimize to much in some cases. So I am thinking of a different scenario, to make both the optimize data range and the verification data range random from a test dataset with a minimum time span, and the two ranges should not overlap more than 50%. Hope this is somewhat in the direction you are thinking of? -Mark http://ayende.com/3678/how-do-you-test-a-system-that-self-optimize-itself#comment5http://ayende.com/3678/how-do-you-test-a-system-that-self-optimize-itself#comment5Thu, 06 Nov 2008 19:14:47 GMTI agree with Frank. I remember in one of our project to test the non deterministic behaviour we used the best case and worst case data samples and we make sure the tests passed all between them. I don't think there is any other way to tackle this problem. The tricky thing here is to find out best case and worst case samples. http://ayende.com/3678/how-do-you-test-a-system-that-self-optimize-itself#comment4http://ayende.com/3678/how-do-you-test-a-system-that-self-optimize-itself#comment4Thu, 06 Nov 2008 17:32:11 GMTDo you need to test the output? If so I would perform the test many times and then run statistics on the data returned. Are 90% of the results > score limit Are all of the results >= lower median-20% http://ayende.com/3678/how-do-you-test-a-system-that-self-optimize-itself#comment3http://ayende.com/3678/how-do-you-test-a-system-that-self-optimize-itself#comment3Thu, 06 Nov 2008 16:54:09 GMTI can't quite see from your description where the self-optimization occurs. Where is the feedback? Does the score limit change, and is that deterministic? If you want to find out how random your randomness is, then there are surely mathematical tools that will allow you to decide how random a selected subset is, if you have the original set and the subset. http://ayende.com/3678/how-do-you-test-a-system-that-self-optimize-itself#comment2http://ayende.com/3678/how-do-you-test-a-system-that-self-optimize-itself#comment2Thu, 06 Nov 2008 16:31:47 GMTHi, What you describe sounds very much like a genetic algorithm based software. The bad new is that testing fuzzy logic system is a bitch. I don't think theres a good way to unit test the optimization process itself. In fact the internal behavior of such systems i.e. what make such algorithm tick, is the subject of many academic research (meaning that no one truly understands it) You can try to measure out the end result of the optimization process by running a known set of test data through the system after the optimization. doing so will at least allow you to make sure that the optimization process reached a satisfactory point. http://ayende.com/3678/how-do-you-test-a-system-that-self-optimize-itself#comment1http://ayende.com/3678/how-do-you-test-a-system-that-self-optimize-itself#comment1Thu, 06 Nov 2008 16:10:26 GMT