1
$\begingroup$

When using Machine Learning for predicting stocks, can a lower Mean Squared Error result in less profit after Backtesting or is there a mistake in the experiment?

$\endgroup$
3
  • $\begingroup$ It can. Good forecast doesn't mean neither big profits or frequent ones. $\endgroup$ – Lisa Ann May 1 '19 at 15:28
  • $\begingroup$ @LisaAnn So the loss function doesn't mean more accuracy? What's the solution? $\endgroup$ – Adb100 May 1 '19 at 15:33
  • $\begingroup$ The problem is that your trading model could not be able to properly exploit your forecast model accuracy. The solution is making your trading model performances more dependent on your forecast model accuracy. $\endgroup$ – Lisa Ann May 1 '19 at 15:41
1
$\begingroup$

My answer to a similar question on the Cross Validated forum link here might be useful. In a nutshell you need to optimise for profit and not MSE - the two are not necessarily one and the same.

$\endgroup$
3
  • $\begingroup$ So your suggestion is a custom loss function? $\endgroup$ – Adb100 May 1 '19 at 15:36
  • $\begingroup$ custom loss is generally very diffucult to implement as it should be smooth/convex function so usual gradient descent methods to work well. PNL is very noisy , therefore training model on PNL would be very diffucult. But it can be used as out of sample benchmark to prefer one model over another. $\endgroup$ – alexprice May 1 '19 at 19:31
  • $\begingroup$ @Adb100 A custom loss function is one solution; others are weighting your training example training losses in proportion to size of move and/or eliminating training examples with moves below a threshold value. $\endgroup$ – babelproofreader May 2 '19 at 15:53
1
$\begingroup$

It can , it depends on your trading strategy.

Let's say model1 predicts relative return as 0.1 for next day and model2 predicts relative return as 0.3, while actual return is 0.15.

Model1 has lower RMSE error.

If your trading strategy is to buy when model predicts positive next day return,and do so in volume proportional to model prediction, then you would buy more asset in model2 case, even if accuracy is lower. but you will make more money in this case. i.e. PNL of model1 would be less than PNL of model2.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.