Take the 2-minute tour ×
Quantitative Finance Stack Exchange is a question and answer site for finance professionals and academics. It's 100% free, no registration required.

I come from a different field (Machine learning/AI/data science), but aim to ask a philosophical question with the utmost respect: Why do quantitative financial analysts (analysts/traders/etc.) prefer (or at least seem) traditional statistical methods (traditional = frequentist/regression/normal correlation methods/ts analysis) over newer AI/machine learning methods? I've read a million models, but it seems biased? Background: I recently joined a 1B AUM (I know it's not a ton) asset management firm. I was asked to build a new model for a sector rotation strategy (basically predicting which SP 500 sector would do the best over 6 months-- chose to use forward rolling 6 month returns) they employ and my first inclination was to combine ARIMA (traditional) with random forest (feature selection) and a categorical (based on normal distribution standard deviation) gradient boosted classifier for ETFs in each sector. Not to be rude, but I beat the ValuLine timeliness for each sector. I used the above mentioned returns as my indicator and pretty much threw everthing at the wall for predictors initially (basically just combing FRED), then used randomForest to select features. I ended up combining EMA and percent change to create a pretty solid model that, like I said, beat ValuLine.

I've read a lot of literature, and I haven't seen anyone do anything like this. Any help in terms of pointing me in the right direction for literature? Or any answers to the overarching idea of why isn't there more machine learning in equity markets (forgetting social/news analysis)? EDIT: For clarification, I'm really interested in long-term predictions (I think Shiller was right) based on macro predictors.


PS- I've been lurking for a while. Thanks for all the awesome questions, answers, and discussions.

share|improve this question
with all due respect, but why would you believe from the start that you have any information or data points at hand that would assist you in predicting which sectors outperform the broad market in 6 months? If I have to summarize my take on financial markets then I would say success has everything to do with managing risk in smart ways as well as seizing opportunities in times of market inefficiencies and it has very little to nothing to do with forecasting the future. –  Matt Wolf Oct 31 '13 at 4:48
I interned at a buy side firm at the start of my career and passed up an offer to join because I felt their effort to rotate company fundamentals millions of times in the hope to discover a predictive multi factor model was fruitless and ridiculous. That was a fund managing close to 100B and everyone went home happy each day knowing they had their guaranteed base and bonuses pocketed and neglecting the fact that they generated returns very similar to their benchmarks. I simply ask why you think metrics that are already priced into assets should have predictive power... –  Matt Wolf Oct 31 '13 at 4:55
You would necessarily beat ValuLine because you fit to the in sample data ... If I was using your research approach I could build a model that has a return at any point in $\mathbb{R}$ of my choosing. –  user2763361 Oct 31 '13 at 11:26
Somehow no one has mentioned that essentially regression methods are subset of ML and can be considered as border cases of various ML algorithms. At first glance this could mean that the results ML provides should always be better, but that's not the case because it is so easy to overfit or bias or simply misuse a more complex technique. While most of ML algorithms are seen as black boxes, it is very easy to produce bad model if you don't know what you are doing. But if you do know what you are doing, certainly they should outperform regression. –  sashkello Oct 31 '13 at 11:43
My understanding is that there is not much ML first of all because a large percentage of people are not very quant-minded at all and prefer experience (or, worse, hunch) and base predictions on fundamental research, or in general are afraid of new things which are not thoroughly tested (and/or they can't understand). The lack of literature in this area is mostly connected with the fact that this is simply a time series (statistical) analysis (not much different from analyzing any other time-series process) and there's more than enough good academic literature on this. –  sashkello Oct 31 '13 at 11:50

3 Answers 3

up vote 19 down vote accepted

Because of:

  • The (extreme) dominance of noise over signal
  • The prevalence of non-repeating patterns (many of which we know are not going to repeat)
  • A pathetic sample size for cross-validation
  • Regime changes due to exogenous events. These are typically in the cross-val window which makes it even worse. (GFC, financial integration, trade law changes, interest rate adjustments by central banks, some idiot in a bank was hiding trades and loses 5 billions dollars, etc).
  • It is well known that non-linear relationships are generally just artefacts of the in sample dataset

There is also the following:

  • Much price changes are driven by news such as a plane crashing or a merger announcement. Are you trying to forecast news (!?) by getting your model to learn non-linear relationships on price data? It should be clear that, if American Airlines price falls due to a terrorist hijacking, it is not going to be useful to have a random forest learn any patterns that result since it will not repeat.

Because of these factors many (econometricians and practicioners) will try to use a priori knowledge to select features and impose constraints on the model in an attempt to improve generalization. This is perceived as necessary by econometricians since the data is too thin, noisy and nonstationary (i.e., the above reasons).

This is not to say that "machine learning" methods such as Lasso, NNG, Elastic Nets or Ridge can't be applied. They result in essentially linear models and you can impose whatever a priori constraints on it through the metaparameters in the loss function or by using a variant that preserves hierarchies when using indicator function interactions (Tibshirani 2013...). Edit: You will still need to select which features go into the algorithm (as a prior imposition) but you can use these to achieve slightly more sparsity than you would otherwise have and introduce some bias into your conditional expectation (or state probability if you're doing multinomial categorical GLM) for improvement in variance of sampling distribution.

I am however open to random forests with the right a priori constraints in place.

There are indeed hundreds of papers that use machine learning to forecast financial markets. Just google something silly like "fuzzy bayesian expert adaptive learners with PSO training S&P 500" and you will get a lesson in the file-drawer effect, publication bias and substandard research methodologies (e.g. selecting 3 of 50 algorithms and 2 of 50 indices and hoping it convinces people).

However, the above is an optimist's view of the industry. From those I've spoken to at low frequency funds they are simply ignorant of machine learning and couldn't apply it because they lack the knowledge and skills. If they were actually interested in being true quants, who knows how much damage they could do with deep learning or something.

If you want to do real machine learning in finance and actually do something that is meritocratic/skill/scientific instead of almost completely random and full of people who practice nonsense, go to a HFT firm (not that most people practice non-sense in low frequency funds, just that many do and this is something that is absolutely impossible to get away with in HFT). That said, I am continually and consistently underwhelmed when I hear of the research methods of low frequency quant funds.

share|improve this answer
Would upvote multiple times if I could. Though, I slightly disagree with your hft claims. I would categorize the "quant IQ" requirements to implement hft algorithms as being very low, on the other hand be a top notch programmer and you most likely make more than any of the quants at a meritocratic hft house. –  Matt Wolf Oct 31 '13 at 14:33
@MattWolf Hmm there is competition both in the algo and in the hardware/software space. Quant IQ is required for the algo development (it is not all simple stuff!, trust me - ultra HFT experiences may differ). There are plenty opportunities for real non-linear machine learning. If it were only about ultra low latency programming then there wouldn't be firms that hire mathematicians, statisticians and the like with little to no object oriented programming experience. –  user2763361 Nov 4 '13 at 16:13
@MattWolf That said, there is no need to be a pen to paper math whiz as may be required (or beneficial) in some derivative pricing roles. What's needed is someone who can easily get top 0.2% on quantitative IQ tests since these are the problems that come up all the time, someone who knows a lot about data mining, and a great programmer. –  user2763361 Nov 15 '13 at 12:54
ML may work well if (I refer to regression ML observing general rules of econometric modeling) - The (extreme) dominance of noise over signal - cleaning precisely noisie from signal and modeling it correctly. - The prevalence of non-repeating patterns (many of which we know are not going to repeat) - Work with a good signal identification method - pathetic sample size for cross-validation - Agree may montecarlo help - non-linear relationships are generally just artefacts of the in sample dataset - overfitting possible with non linear ML but precise signal and noise identification may help. –  Barnaby Apr 17 at 0:14

I was going to comment but it turned out to be quite elaborate.

My experience with certain AI/ML methods is that they're not deterministic. Take RBM for instance, a very wide-spread paradigm. To train such a machine you have two approaches, backpropagation or Kullback-Leibler divergence. Both require you to initialise the machine to a random state. And that makes them non-deterministic.

Even more problematic for instance are methods like Simulated Annealing or Genetic Programming to find maxima or minima in a landscape where every single step contains a non-deterministic component.

In general there's nothing wrong with a hint of non-determinism, especially in our racket where nearly everything is a stochastic process. It just becomes a problem when you need a reproducible result, for instance when comparing a live session with some backtest. It's just very hard to decompose tracking errors (e.g. divergence between paper session and live session) into systemic errors and the divergence of, say, the random generators (or more general the sources of non-determinism).

Conversely, if you try to turn a non-deterministic algorithm into a deterministic one by using a fixed (or deterministic) sequence of ``non-determinism'' the algorithm will collapse to a variant that's probably less efficient than a good deterministic one (e.g. deterministic SA vs. greedy search); of course I can't prove that here but if I had to I'd show that the probability to pick a sequence of randomness out of all possible sequences of randomness that reflects exactly the invariants before or after each step of a greedy round is abyssmally small.

share|improve this answer
Anyone who attempts to generate random numbers by deterministic means is, of course, living in a state of sin. -- John von Neumann Your answer does not explain to me why you could not take a fixed seed when using these methods. –  Bob Jansen Oct 31 '13 at 10:53
@BobJansen Well technically you can, but you will see other artefacts like an SA algorithm never finding a global maximum; or if you start on your landscape at a slightly different position but with the same sequence of ``non-determinism'' you might not find the same maximum that you used to find –  hroptatyr Oct 31 '13 at 11:02
This is certainly not the reason why machine learning is rarely utilised at low frequency quant funds. –  user2763361 Nov 2 '13 at 16:42

It's probably because of the strong long-standing statistical underpinnings in economics and econometrics, and overall, risk prediction. For example, look at current research with fat-tail distributions and calculations for Expected Tail Loss (ETL), etc. These studies fit Student's t, Normal, Stable, and Pareto probability distributions to data and report that e.g. the Kolmogorov or Anderson-Darling goodness-of-fit distance is greater for the normal distribution, that is, the normal distribution doesn't recover the area in the tails as well as the Stable and Student's t (for varying d.f.). Next, moving into time-series analyses, there is still tremendous merit in using ARIMA, ARMA, ARCH, and GARCH-type fitting to explaining conditional means and variances, and autocorrelation. You can simply use a handful of the methods listed above and perform an incredible amount of risk prediction that has scientific merit in financial risk management.

Next, for ML, the more you delve into non-linear manifold learning (ISOMAP, Laplacian eigenmaps, etc), metaheuristics (evolutionary algorithms and evolutionary strategies such as genetic algorithms, covariance matrix self-adaptation, ant colony optimization, particle swarm optimization), neural adaptive learning (ANNs), and many other ML and AI supervised methods you are essentially getting further away from deterministic and stochastic derivative-based gradient descent and Newton-Raphson -- which provide parameter uncertainty.

Recently, I started using particle swarm optimization for almost everything I optimize. Sure, Newton-Raphson is faster, is more consistent, and can provide parameter uncertainty, but as I see it, to correctly model risk one needs to combine realizations form many different distributions to build up a final uncertainty distribution. For this reason, the majority of time when I am not interested in Type I and II errors, power, AIC, etc, or for that matter the standard errors of coefficients, I typically spew out point estimates, and then use these as inputs to Monte Carlo analysis.

Regarding random forests (RF), it is one of the best unsupervised and supervised classifiers out there. RF simultaneously combines bootstrapping the training sample and randomly selects features to train each tree. Unknown test objects not in the training sample are then "dropped" down each train tree. Feature importance is then determined by permuting value of feature j in the out of sample objects and dropping them down the same trained tree. Breiman suggested using at least 5000 trees ("don't be stingy") and in fact his main ML paper on RF used 50,000 trees per run.

Artificial neural networks (ANNs) have been shown be a universal approximator, however, they become quite expensive when compared with Newton-Raphson, providing you know the objective function your are fitting (optimizing). If you don't know the objective equation, then an ANN can be tremendously beneficial. Like I have said to other colleagues, in these cases, "the ANN is the equation." For this latter framework, where the ANN is the equation, if you draw random quantiles from distributions for various risk factors, and clamp each set of quantiles to the input nodes of the ANN, train using CV, the utility of an ANN combined with Monte Carlo will be immense.

Since it takes a decade or more to learn ML and AI at a level where you can tackle any problem, there would be no expectation that someone with an extensive QF background would be able to pick it up quickly, and no expectation that the results would be better. ML and AI are best implemented for near NP-hard problems such as timetabling, shift and airline scheduling, logistics, where there is not a closed form analytic solution (equation) which describes your model.

share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.