In an interview from '96 Bill Eckhardt points out that there are tests that any system must pass to be taken seriously. That is: tests for (1) overfitting, (2) post-dictiveness, (3) maldistribution of returns, (4) statistical artifacts and (5) the degree to which a system takes advantage of unusual and possibly nonrepeatable circumstances.

I'm aware of tests like White's RC or Hansen's (stepwise) SPA for data snooping..but there are obviously some testing procedures I haven't come across so far.

So which tests can be used to deal with the issues listed above?


3 Answers 3


I would add the Monte Carlo position permutation test to your list (see here for more details and book). The null in White's RC is that the system's returns are zero, but in the position permutation test the null is that the system's positioning (long, short or out of the market) is no better than random.

Incidentally, there is an R package, ttrTests, which implements White's RC and Hansen's SPA, along with other useful tests.

Edit: Have also thought of the "torture test." (see my answer here) If an otherwise acceptable system seriously degrades when subjected to this test, it shows that the system is highly reliant on getting good entries and/or exits.

  • $\begingroup$ I agree, the MC position permutation is a valuable test one might consider when evaluating a system. And the ttrTests package also seems to be quite useful - unfortunately it doesn't include the stepwise SPA test, which is supposed to be more accurate than the RC and the (ordinary) SPA test. $\endgroup$
    – user2278
    Apr 11, 2012 at 15:59

To put your above mentioned points into perspective you should definitely consider this seminal paper from Attilio Meucci which develops a general framework for trading systems:

‘The Prayer’ Ten-Step Checklist for Advanced Risk and Portfolio Management

From the abstract:

"We present “The Prayer”, a recipe of ten sequential steps for all portfolio managers, risk managers, algorithmic traders across all asset classes and all investment horizons, to model and manage the P&L distribution of their positions.

For each of the ten steps of the Prayer, we introduce all the key concepts with precise notation; we illustrate the key concepts by means of a simple case study that can be handled with analytical formulas; we point the readers toward multiple advanced approaches to address the non-trivial practical problems of real-life risk modeling; and we highlight a non-exhaustive list of common pitfalls."


The best way IMO to validate a system is to walk the system forward, periodically re-optimizing. This will quickly tell you if you are over-fitting, the usual metrics can then be applied to the generated results.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.