What advanced statistical techniques are quant researchers using? [closed]

Question

Some time ago I was talking to a representative of one of the leading quantitative hedge funds. We talked about recruitment and the following had me stunned:

The representative explained that they used to hire graduate statisticians and PhDs from Maths, Physics, CS but have not done so in recent years. They claimed that they found that the statistical techniques the firm was using were too advanced and that the learning curve for new hires from that category was too step. Unfortunately they were not prepared to give examples.

Can somebody explain to me what techniques they might have been referring to? I am not looking for detailed insights into the strategies but rather the "toolbox" that firms use that would be too advanced for graduate statisticians or physics PhDs.

Would graphical models fall in that category?

Answer 1

I spoke to a quant trader from one of the leading quant proprietary trading firms about a year ago. Apparently, there was a point in time when they looked into applying machine-learning techniques to modelling problems. That would include things like neural networks, SVRs, graphical models and a bunch of other related stuff. These are essentially a kind of statistical approach much like the more mainstream quantitative models but arguably more advanced. However, they have since lost interest in this direction due to various difficulties.

These machine-learning approaches have their roots in the engineering and computer science domain in areas like computer vision, controls and AI. The problems dealt with in these fields are of a different nature as compared to those in modelling financial markets and thus really requires a different paradigm. Too many people who attempt to apply these techniques to finance does it blindly without examining whether the underlying assumptions are suitable for markets, and are simply using them "out of the box". Essentially these quants have poor understanding of how these tools work and cannot apply them well.

The fact that many of these machine-learning techniques are presented as black-box models(they aren't really total black-boxes) don't help either. No decent firm in their right mind will place their trust in a black-box model when conventional methods are better understood and open to examination -- which is the whole point of quant trading anyway, to apply the scientific method to markets.

This is not to say that machine-learning can't be used for such purposes, I believe there are firms and people that have found success with these methods. However this requires people with the proper expertise, which is rare. It is much easier to find people with working knowledge of conventional statistical techniques than those who are able to properly apply machine-learning. When you can't get good people, you can't build up the proper expertise in your firm and you're better off pursuing other ways.

I think these difficulties however are not something inherent to machine-learning but because it is still in its infancy in this domain. Orthodox quantitative finance encountered its equivalent of these issues in its beginning as well but became more accepted as things became more widely understood. Machine-learning in finance is a promising direction and I think there's a good chance its potential will be realized in the near future.

Answer 2

I'm converting my comment into an answer because, as far as I can tell from your question, it's an explanation based on "real world" experience with managers....

What I've found, in writing analytics software, is that many analysts "abuse" statistics to form invalid conclusions. I presume that when trained statisticians (or well published algorithms) disagree with their conclusions that the managers assume their methods are too advanced... when, in reality, they're just fallacious.

My theory is that many managers have avoided bringing in advanced mathematicians because they don't want applied math to expose just how much risk they're taking. They'd rather just call it alpha.

Answer 3

I believe your conversation counter party did not mean to say that the theoretical nature of the applied statistical techniques are too advanced for a graduate math/stats/CS level student to master. I venture to assume what he/she meant was that the applied tool sets specifically to financial trading at that specific firm are/were too advanced for someone fresh out of school to learn and apply in a short period of time.

Actually, I believe this to be sometimes true as well, in case this is what he/she meant. First of all, I think the term and stigma attached to "PhD" level kind of work is hopelessly overrated. A PhD program, dissertation, and defense, means nothing else but that the candidate/holder has specialized on a topic and performed studies/reading in order to familiarize him/herself with the topic in order to do work on his/her own that deepens the understanding of the community which applies similar techniques. Having said that, one does not have to undergo such program in order to amass profound knowledge and add value by performing work in a particular area on his/her own. I have seen guys without college degree but an amazing knowledge (or should I say insight) of mass psychology and expertise in market intrinsics achieving promotion after promotion at major trading desks because of their superior return performance.

Trading and also quant research requires a basic/advanced set of tools in order to understand the subject matter. But real success, from my experience, does not correlate with which school or academic program one underwent but highly correlates with an extraordinary passion for the particular job (as with everything in life), constant learning on the job and from others, an insight into how market players think and behave, time spent to study market structure, a healthy risk appetite but above all a deep respect and understanding of the risk-reward relationship.

Such skills and learned knowledge can often be as complex if not more complex than what PhD level students study and expose themselves to in the ivory towers of the world. Those skills and abilities, imho, often are lacking in many students who pursued relatively theoretical work in university. I loved to ask incoming PhD or science grad school students at my previous trading desks on the sell side what strategy ideas they had, what they think may be worthwhile pursuing, research wise. I asked them what ideas they would pitch to me on the spot. Most of the time there was a shocking silence afterwards and I immediately knew that such student hardly at all spent time on applying any of his/her knowledge to financial markets/time series/assets before.

This bridge/gap, imho, is what is much harder to overcome than learning the tool sets required to study and research financial markets. Additionally, many of such candidates lack a commercial way of thinking, they believe they are paid to perform research like the 1960s/1970s Bell lab guys. The life cycle of what works and does not work in finance is much shorter than in other disciplines and that is something that is very hard to teach and punch into a PhD who comes along with a somewhat academic arrogance, thinking he can conquer the world. Personally I have met countless top traders from universities I have hardly ever heard about, while I most never came across (personally) someone with a highly specialized advanced degree from a top school who beat the rest of the pack.

Long story short, I believe that is what your representative may have meant and referred to. Knowing the theoretical tool is one thing, applying them to finance and extracting alpha requires a much vaster knowledge and is an entirely different thing.