When to stop training?

Question

I have built a deep reinforcement learning based portfolio optimisation agent. At a high level it is using macro economic data, valuations of the assets and a few technical indicators as the features. The policy network is a temporal convolution network with attention. The output of the policy network is the portfolio allocation. The agent is episodic with 365 days as the episode length. The episode terminates either at the end of of 365 days or when the drawdown of the portfolio hits 30%. With these conditions I have trained the agent on 10 years of data. The problem I am facing is of figuring when to stop training the agent. All the metrics like loss continue to improve with each iteration. The out of sample performance fluctuates in a narrow band after some 100 iterations indicating that it is possibly the best result the model can provide. But the loss metric keep decreasing.

As one can see from these graphs that one cannot make a decision on when to stop the training based on the training loss. And any decision cannot be made based on Out of Sample Validation data set. Any suggestions ?

Answer 1

The standard approach for cross-validation is to work with three datasets

• the training set,
• the validation set,
• the test set.

You train on the training set (no surprise), and you use the validation set to decide for your hyper-parameters. Then you retrain, test on the test set (no surprise again), and it is done.

My advice here is to consider that the number of epochs to train on is a hyper-parameter: thus choose it to generalize well on your training set. Then retrain with it and it is done.

One more detail: splitting the history in sets is not trivial when we talk about time-series. You should take "contiguous blocks"; if you have 10 years (this is short, moreover you do not have the 2008 crisis in your history!)

• 6 years for training,
• 1.5 years for validation
• the last 1.5 years for the test (this is very short...)

You can use K-fold cross-validation to try to obtain more robust results: k-fold your 7.5 years by rotating the 1.5 years you keep for validation. Thanks to this you will have 5 validation sets to look at.

Answer 2

One option could be to use an early stopping mechanism, e.g. stop when the average reward per episode on a validation set stops increasing. In practice, I found this to be a bad idea as you generally don't have enough data to be able to afford a validation set when using daily data (in addition to the test set). If you are dealing with higher frequency data then daily, the story might be different.

A second option is to use a simple policy network (e.g. a linear regression) which does not have enough capacity to overfit, and so you can train until when the reward on the training set stops to increase.

A third option is to train for long enough (i.e. until when the reward stops increasing or until when you are confident to be in the overfitting region) while checkpointing the agent during training. Then use visualizations to inspect the behavior of the agent during training (e.g. Sharpe ratio or turnover as a function of the number of training steps). This is qualitative which doesn't tell you when to stop, but it does provide insights /let a human judge what could be a decent stopping point (e.g. turnover will probably become too high after a certain number of iterations)

In addition, this problem should be evaluated in the context of walk-forward optimization, just to add another dimension to this already difficult problem.