Quantitative Finance Stack Exchange is a question and answer site for finance professionals and academics. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

In an interview from '96 Bill Eckhardt points out that there are tests that any system must pass to be taken seriously. That is: tests for (1) overfitting, (2) post-dictiveness, (3) maldistribution of returns, (4) statistical artifacts and (5) the degree to which a system takes advantage of unusual and possibly nonrepeatable circumstances.

I'm aware of tests like White's RC or Hansen's (stepwise) SPA for data snooping..but there are obviously some testing procedures I haven't come across so far.

So which tests can be used to deal with the issues listed above?

share|improve this question

I would add the Monte Carlo position permutation test to your list (see here for more details and book). The null in White's RC is that the system's returns are zero, but in the position permutation test the null is that the system's positioning (long, short or out of the market) is no better than random.

Incidentally, there is an R package, ttrTests, which implements White's RC and Hansen's SPA, along with other useful tests.

Edit: Have also thought of the "torture test." (see my answer here) If an otherwise acceptable system seriously degrades when subjected to this test, it shows that the system is highly reliant on getting good entries and/or exits.

share|improve this answer
I agree, the MC position permutation is a valuable test one might consider when evaluating a system. And the ttrTests package also seems to be quite useful - unfortunately it doesn't include the stepwise SPA test, which is supposed to be more accurate than the RC and the (ordinary) SPA test. – user2278 Apr 11 '12 at 15:59

To put your above mentioned points into perspective you should definitely consider this seminal paper from Attilio Meucci which develops a general framework for trading systems:

‘The Prayer’ Ten-Step Checklist for Advanced Risk and Portfolio Management

From the abstract:

"We present “The Prayer”, a recipe of ten sequential steps for all portfolio managers, risk managers, algorithmic traders across all asset classes and all investment horizons, to model and manage the P&L distribution of their positions.

For each of the ten steps of the Prayer, we introduce all the key concepts with precise notation; we illustrate the key concepts by means of a simple case study that can be handled with analytical formulas; we point the readers toward multiple advanced approaches to address the non-trivial practical problems of real-life risk modeling; and we highlight a non-exhaustive list of common pitfalls."

share|improve this answer

The best way IMO to validate a system is to walk the system forward, periodically re-optimizing. This will quickly tell you if you are over-fitting, the usual metrics can then be applied to the generated results.

share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.