I have the following strategy pipeline which is a function of several hyperparameters and execution parameters:

for each instrument {
    1. Calculate features
    2. Split the data into (training, validation, test) // test not used yet

    for each hyperparameter permutation {
        3. Fit regression model on training, calculate predictions on validation
        4. Use validation predictions to simulate pnl

    5. Choose hyperparameters that maximize f(pnl) // Sharpe, etc

6. Add best performers to portfolio
7. Evaluate portfolio performance on test set

Can anyone find anything wrong with this process? In the past, I did a training / validation split without a final test set, and ended the process at step 6. The consequence was that my optimized portfolio did not perform as expected during live testing. In other words, the portfolio selection process didn't generalize, and the top performers during the validation period did not perform well during live testing. By adding step 7, I aim to know about such failures before going into live testing.

  • $\begingroup$ How is the splitting done? $\endgroup$ – Matthew Gunn Jan 19 '18 at 19:07
  • $\begingroup$ Splitting is done chronologically: [training, validation, test], with test being most recent. I'm using 0.6, 0.2, 0.2 as their respective ratios. $\endgroup$ – tmakino Jan 19 '18 at 19:31

Some people claim (and I tend to agree with them) that splitting data into [training, validation, test] subsets may not be a right approach at all when backtesting trading strategies. The alternative way is to backtest by retraining your model at every historical day/bar using past data and taking a prediction for the next day/bar (kind of walk-forward but retraining as often as possible) and after all measure DMB (data mining bias - very important!) instead of performing the out-of-sample tests. Then for live trading you select only strategies having low DMB values. There is an old, but very nice thread at FF which gives more insights into this methodology: https://www.forexfactory.com/showthread.php?p=7927988#post7927988

  • $\begingroup$ Testing the entire strategy pipeline on bootstrapped random data is a good idea. $\endgroup$ – tmakino Jan 20 '18 at 18:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.