I'm building stock selection models, and pick top 5 and bottom 5 stocks. Given the variability in Stochastic gradient decent results, they keep changing. One way to get consistent results is to use the random seed, but I'm looking for if there a better way to deal with this. Also how would you interpret the results, i.e. One set of top 5 versus another set of Top 5 picks (3-4 of them are the same, but may differ in ranking). I'm running enough iterations to know this isn't an issue about convergence.
Have you tried to choose an arbitrary number of model, let say 20, each one having its own seed? Then you run your twenty models and use the median of your 20 results as signal. One advantage of that method is that you can also get a confidence estimate of your prediction thanks to the standard deviation of your 20 results.
The best bet for you is to use Ensemble Learning, as someone experienced with Kaggle competitions, the best way to replicate good performance on Private Learderboard is to ensemble as many algorithms together. This includes intra and inter ensembling. Intra meaning ensembling same algorithms (e.g Xgboost) but with different tuning parameters. You can chose top 10 parameters by cross-validation results to intra-ensemble. Some participants also intra-ensemble different random seeds of same parameters, taking total number of models to more than 500! Second is inter-ensembling, in this you would ensemble different algorithms (e.g neural net, random forest and xgboost), the way you choose these algorithms is by looking at two things : 1) The cross-validation accuracy for each algorithm should be nearby, 2) The correlation in cross-validation predictions should not be more than 80-90%
Stochastic solutions are an unavoidable property of stochastic methods, in particular optimisation methods. See for instance section 3 in A Review of Heuristic Optimization Methods in Econometrics. In general, you cannot get rid of randomness; you need to analyse it, by looking at and analysing distributions (e.g. of portfolios) instead of single numbers. See for instance An Empirical Analysis of Alternative Portfolio Selection Criteria (of which I am a coauthor).
Convergence means (at best) that the algorithm has stopped in a local optimum. If you have multiple optima, the algorithm may stop at different optima. Have you compared the objective-functions values of repeated runs? Even if they are the same: it means that the algorithm, or more specifically, your selection criterion (=objective function) cannot differentiate between different solutions. Could you modify the model, e.g. add more constraints?