I have a system in which one of the core parts is required to work in a non deterministic fashion. In particular, some part of the system is behaving randomly (by design). Here is a small description:
- 90% of the time, we select only items that are above the score limit.
- Selection between those is made randomly, with no bias
- 10% of the time, we select only items that are _below_ the score limit.
- Selection among those is done based on the oldest last shown date
- To skip "rotten apples", we calculate the median of the lowest group and exclude all items that are below 20% of the median.
The idea behind this is to have a bare bone system that can adapt in response to input.
But I am not sure how I can test this in a way that will actually reveal something meaningful about the system.