How do you test a system that self optimize itself?

time to read 1 min | 154 words

I have a system in which one of the core parts is required to work in a non deterministic fashion. In particular, some part of the system is behaving randomly (by design). Here is a small description:

  • 90% of the time, we select only items that are above the score limit.
    • Selection between those is made randomly, with no bias
  • 10% of the time, we select only items that are _below_ the score limit.
    • Selection among those is done based on the oldest last shown date
    • To skip "rotten apples", we calculate the median of the lowest group and exclude all items that are below 20% of the median.

The idea behind this is to have a bare bone system that can adapt in response to input.

But I am not sure how I can test this in a way that will actually reveal something meaningful about the system.