How to properly set strategy parameters and select portfolio

Question

I have the following strategy pipeline which is a function of several hyperparameters and execution parameters:

for each instrument {
    1. Calculate features
    2. Split the data into (training, validation, test) // test not used yet

    for each hyperparameter permutation {
        3. Fit regression model on training, calculate predictions on validation
        4. Use validation predictions to simulate pnl
    }

    5. Choose hyperparameters that maximize f(pnl) // Sharpe, etc
}    

6. Add best performers to portfolio
7. Evaluate portfolio performance on test set

Can anyone find anything wrong with this process? In the past, I did a training / validation split without a final test set, and ended the process at step 6. The consequence was that my optimized portfolio did not perform as expected during live testing. In other words, the portfolio selection process didn't generalize, and the top performers during the validation period did not perform well during live testing. By adding step 7, I aim to know about such failures before going into live testing.

Answer 1

Some people claim (and I tend to agree with them) that splitting data into [training, validation, test] subsets may not be a right approach at all when backtesting trading strategies. The alternative way is to backtest by retraining your model at every historical day/bar using past data and taking a prediction for the next day/bar (kind of walk-forward but retraining as often as possible) and after all measure DMB (data mining bias - very important!) instead of performing the out-of-sample tests. Then for live trading you select only strategies having low DMB values. There is an old, but very nice thread at FF which gives more insights into this methodology: https://www.forexfactory.com/showthread.php?p=7927988#post7927988