Tests that any system must pass to be taken seriously

Question

In an interview from '96 Bill Eckhardt points out that there are tests that any system must pass to be taken seriously. That is: tests for (1) overfitting, (2) post-dictiveness, (3) maldistribution of returns, (4) statistical artifacts and (5) the degree to which a system takes advantage of unusual and possibly nonrepeatable circumstances.

I'm aware of tests like White's RC or Hansen's (stepwise) SPA for data snooping..but there are obviously some testing procedures I haven't come across so far.

So which tests can be used to deal with the issues listed above?

Answer 1

I would add the Monte Carlo position permutation test to your list (see here for more details and book). The null in White's RC is that the system's returns are zero, but in the position permutation test the null is that the system's positioning (long, short or out of the market) is no better than random.

Incidentally, there is an R package, ttrTests, which implements White's RC and Hansen's SPA, along with other useful tests.

Edit: Have also thought of the "torture test." (see my answer here) If an otherwise acceptable system seriously degrades when subjected to this test, it shows that the system is highly reliant on getting good entries and/or exits.

Answer 2

To put your above mentioned points into perspective you should definitely consider this seminal paper from Attilio Meucci which develops a general framework for trading systems:

‘The Prayer’ Ten-Step Checklist for Advanced Risk and Portfolio Management

From the abstract:

"We present “The Prayer”, a recipe of ten sequential steps for all portfolio managers, risk managers, algorithmic traders across all asset classes and all investment horizons, to model and manage the P&L distribution of their positions.

For each of the ten steps of the Prayer, we introduce all the key concepts with precise notation; we illustrate the key concepts by means of a simple case study that can be handled with analytical formulas; we point the readers toward multiple advanced approaches to address the non-trivial practical problems of real-life risk modeling; and we highlight a non-exhaustive list of common pitfalls."

Answer 3

The best way IMO to validate a system is to walk the system forward, periodically re-optimizing. This will quickly tell you if you are over-fitting, the usual metrics can then be applied to the generated results.