# How to properly set strategy parameters and select portfolio

I have the following strategy pipeline which is a function of several hyperparameters and execution parameters:

for each instrument {
1. Calculate features
2. Split the data into (training, validation, test) // test not used yet

for each hyperparameter permutation {
3. Fit regression model on training, calculate predictions on validation
4. Use validation predictions to simulate pnl
}

5. Choose hyperparameters that maximize f(pnl) // Sharpe, etc
}

6. Add best performers to portfolio
7. Evaluate portfolio performance on test set


Can anyone find anything wrong with this process? In the past, I did a training / validation split without a final test set, and ended the process at step 6. The consequence was that my optimized portfolio did not perform as expected during live testing. In other words, the portfolio selection process didn't generalize, and the top performers during the validation period did not perform well during live testing. By adding step 7, I aim to know about such failures before going into live testing.

• How is the splitting done? – Matthew Gunn Jan 19 '18 at 19:07
• Splitting is done chronologically: [training, validation, test], with test being most recent. I'm using 0.6, 0.2, 0.2 as their respective ratios. – tmakino Jan 19 '18 at 19:31