Validation set on Walk Forward Analysis

Question

When backtesting a trading strategy using Walk Forward Analysis/Optimization, I see people split each window into training and testing sets.

Suppose you want to select the best combination of MAs for your MACD strategy, you would first try the different combinations on your training set, select the best one, and report the results you get with those parameters on your test set.

However, when the strategy in place is a machine-learning model, shouldn't we have a validation set as we have in every other ML problem? The training set would be for learning the model parameters, the validation set to select the best-performing hyper-parameters, and finally, the test set to report the model results.

I haven't seen anywhere where a validation set is used in WFA/WFO. I would appreciate any input or extra resources on this topic.

Cheers!

Answer 1

You should treat backtesting strategies exactly as you would treat validating a machine learning model — look for ways to maximise your out-of-sample testing periods.

This means train-test-validation sets, with the validation being totally aside from your iteration and strategy development process, a sort of final crucible for your strategy.

As always, take the back test with a pinch of salt, it is much more important to reconcile your views of the forward path of markets with the underlying inefficiency the strategy is built to exploit - this is where most machine learning tools fail as the underlying inefficiency the model is apparently learning is somewhat unclear and thus it is unclear when the models performance will start to decay.