In Marcos Lopez de Prado's book, Advances in financial machine learning, he recommends using the combinatorial purged cross-validation(CPCV) for backtesting. His motivation is sensible. Through the method, we can test our trading strategy with more scenarios and we can reduce the false discovery rate.

Our training set is not continuous anymore with his CPCV method. As an example, our training set may combine the two time series, A(20018-01-01~2019-01-01) and B(2019-07-01~2020-01-01). We would have a big price jump at the time point where A and B are connected as there is a 6 months gap.

But with his cross-validation method, we consider the time series as one continuous time series and calculate all quantities. We will have discontinuities of quantities such as features, target labels, indicators, etc at the point. These discontinuities in the quantities bother me.

The discontinuities are artificial at most. I don't think we will meet the price jump in real-life and I feel the scenarios generated by CPCV are not the real one. Why do we bother to check our model in the unreal scenarios?

Or do we need to take an additional preprocessing step to make the price continuously? (I doubt this though as Lopez didn't mention the step at all.)

  • $\begingroup$ Imagine two groups of people (or computers) at a hedge fund. Every day the first group of people performs all necessary calculations and issues recommendations (buy/sell etc). This group experiences no discontinuity. The second group of people receives the recommendations and can act on them (which will cause a profit or loss of real money). On the first day of a "holdout period" the second group exits existing positions and refuses to implement the recommendations (they go to the beach instead). They resume trading (and making P&L) on the first day of the next "evaluation period". $\endgroup$ – noob2 May 27 at 17:14
  • $\begingroup$ How does this supposed analogy mirror the CPCV method and its discontinuity issue? $\endgroup$ – develarist Jun 15 at 10:48
  • $\begingroup$ @develarist What I mean is that the scenarios generated by CPCV might not be encountered in real life. $\endgroup$ – kevin012 Jun 15 at 12:15
  • $\begingroup$ i wasn't commenting on ur post but the reply to ur post? $\endgroup$ – develarist Jun 15 at 12:19
  • $\begingroup$ @develarist sorry. I misunderstood. $\endgroup$ – kevin012 Jun 15 at 12:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.