2
$\begingroup$

I'm building stock selection models, and pick top 5 and bottom 5 stocks. Given the variability in Stochastic gradient decent results, they keep changing. One way to get consistent results is to use the random seed, but I'm looking for if there a better way to deal with this. Also how would you interpret the results, i.e. One set of top 5 versus another set of Top 5 picks (3-4 of them are the same, but may differ in ranking). I'm running enough iterations to know this isn't an issue about convergence.

$\endgroup$
1
$\begingroup$

Have you tried to choose an arbitrary number of model, let say 20, each one having its own seed? Then you run your twenty models and use the median of your 20 results as signal. One advantage of that method is that you can also get a confidence estimate of your prediction thanks to the standard deviation of your 20 results.

| improve this answer | |
$\endgroup$
  • $\begingroup$ is there a source you could cite? $\endgroup$ – user23564 May 26 at 20:32
  • $\begingroup$ @user23564 this technique is used in a paper published by a quant team from a major bank however I am not allowed to cite as the article is for clients only. Besides that I think it’s intuitive enough... $\endgroup$ – SachaTheBrave May 26 at 23:33
1
$\begingroup$

Stochastic solutions are an unavoidable property of stochastic methods, in particular optimisation methods. See for instance section 3 in A Review of Heuristic Optimization Methods in Econometrics. In general, you cannot get rid of randomness; you need to analyse it, by looking at and analysing distributions (e.g. of portfolios) instead of single numbers. See for instance An Empirical Analysis of Alternative Portfolio Selection Criteria (of which I am a coauthor).

Convergence means (at best) that the algorithm has stopped in a local optimum. If you have multiple optima, the algorithm may stop at different optima. Have you compared the objective-functions values of repeated runs? Even if they are the same: it means that the algorithm, or more specifically, your selection criterion (=objective function) cannot differentiate between different solutions. Could you modify the model, e.g. add more constraints?

| improve this answer | |
$\endgroup$
  • $\begingroup$ when you mean constrains, would it be just in terms constrains of the hyperparameters? I wanted to ensure that I dont 'stop too early, so I had increased the model runs significantly with a good amount of variance in the initial guesses. I will run the model with different stopping criteria's as well, see how that affects the model. $\endgroup$ – user23564 Jun 1 at 18:03
  • $\begingroup$ No, I meant constraints in your model; such as diversification constraints. That might be helpful if your different solutions actually map to the same objective function value. $\endgroup$ – Enrico Schumann Jun 2 at 12:27
1
$\begingroup$

The best bet for you is to use Ensemble Learning, as someone experienced with Kaggle competitions, the best way to replicate good performance on Private Learderboard is to ensemble as many algorithms together. This includes intra and inter ensembling. Intra meaning ensembling same algorithms (e.g Xgboost) but with different tuning parameters. You can chose top 10 parameters by cross-validation results to intra-ensemble. Some participants also intra-ensemble different random seeds of same parameters, taking total number of models to more than 500! Second is inter-ensembling, in this you would ensemble different algorithms (e.g neural net, random forest and xgboost), the way you choose these algorithms is by looking at two things : 1) The cross-validation accuracy for each algorithm should be nearby, 2) The correlation in cross-validation predictions should not be more than 80-90%

| improve this answer | |
$\endgroup$
  • $\begingroup$ While this seems to be a good answer to the problem, I think it also introduces an additional complication of interpretability of the model. Any thoughts on that? I could see one using feature importance, to help a bit. $\endgroup$ – user23564 Jun 1 at 17:55
  • 1
    $\begingroup$ I think interpretability will hardly there even if you use a single algorithm like Neural net or Xgboost or random forest, many of the popular ML algorithms aren’t very interpretable, they are Black Box by nature, but that doesn’t mean your investment strategy using them has to be blackbox, if you start with a good fundamental thesis for stock picking and use the ML algos as a tool rather than base framework, you should be good. $\endgroup$ – Dhruv Mahajan Jun 3 at 13:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.