This is an issue I've been struggling with for quite a while, and haven't found a satisfactory answer yet. The basic problem is this:
Lets say you set up a black box system which gives you anywhere from 5-25 opportunities per day. Now lets say you observe x "paper" transactions, and you hit your target on y% of occasions.
Given the above:
- How many observations should you make at minimum before using the system?
- What is the best way to confirm that the edge is more than luck? (i.e. statistical vs. practical significance)
- How will you know if you lose your edge over time versus what might be ordinary draw down or otherwise randomness?
- How does this relate to maximizing growth using the Kelly Criterion?