When using Machine Learning for predicting stocks, can a lower Mean Squared Error result in less profit after Backtesting or is there a mistake in the experiment?
My answer to a similar question on the Cross Validated forum link here might be useful. In a nutshell you need to optimise for profit and not MSE - the two are not necessarily one and the same.
It can , it depends on your trading strategy.
Let's say model1 predicts relative return as 0.1 for next day and model2 predicts relative return as 0.3, while actual return is 0.15.
Model1 has lower RMSE error.
If your trading strategy is to buy when model predicts positive next day return,and do so in volume proportional to model prediction, then you would buy more asset in model2 case, even if accuracy is lower. but you will make more money in this case. i.e. PNL of model1 would be less than PNL of model2.