Thoughts on how quantitative hedge funds use machine learning to invest in the stock market (algorithms, examples of data, etc.)

Question

I believe there are several post on this general topic but I thought I would start my own thread. I'm a former fundamental hedge fund investor (i.e. modeling a company's financials, forecasting the cashflows and discounting them back at a discount rate to get the valuation) who is interested in learning more about Data Science/AI/Machine learning and how it is used in investing. First off, let me assure some of the people who have posted on different threads that this Data Science/AI/Machine Learning does work. The best hedge funds are actually quantitative hedge funds like Renaissance Technologies, DE Shaw, Two Sigma, and I believe they do this type of work. Before the quantitative investors there were the fundamental investors (who spoke with the company, modeled the financials, qualitatively decided if this was a good investment, etc. i.e. like Warren Buffet) and the technical investors/traders who looked at charts and based on certain movements/volumes of the stock chart, they would invest. Frankly I always thought technical investing was nonsense but a lot of people do it and maybe it works maybe it doesn't. Then came the quants but I believe they were initially using complex mathematical models like Black-Scholes pricing models. Now AI/Machine Learning is what the best quantitative hedge funds are using. For my own edification, I'd like to get people's thoughts on how hedge funds are actually employing AI/Machine Learning to investing.

Are there certain algorithms that are better for investing in the stock market?

What types of data is most useful?

Are there different models for predicting short-term movements and long-term price movements?

Being an investor with 10+ years of experience, I have some general high level thoughts but I'm neither a programmer or a data science. Please correct me if I'm wrong. Essentially, these quantitative hedge funds are trying to find patterns in the stock/stocks/other asset classes (gold, bonds, currency, etc.) that have yet to be exploited. As with the other types of investors, they want to know what will make a stock (or any other asset class) go up or down (most hedge funds both long and short stocks). But while the other types of investors have a particular dogma they are following, these guys just care about the "signal" that will indicate whether the stock will go up or down. Stocks and any other asset class go up and down based on supply and demand. Are there more buyers vs. sellers - then the stock goes up. Are there more sellers vs. buyers, then the stock goes down. So in essence, are these quant funds really just modeling/predicting human behavior and taking advantage of their ability to crunch data are a much large and faster capacity? If that's the case, then from a fundamental perspective, I can easily hypothesize what type of data might move a stock/asset class.

For example, if a stock has a certain trading pattern that indicates technical investors will likely buy it over a certain time frame, wouldn't that work? Would there be a certain algorithm that works best for this type of analysis?

Another example might be, if companies that exhibit certain revenue growth metrics, or margin improvement, would that signal a potential buy opportunity? Or perhaps if certain words in their annual report, quarterly filings, press releases indicate this company is likely to do well?

Or perhaps the holders list (the type of investor that owns the stock) could provide an indication of how the stock might react in the future.

I could go on but I might be totally off base. I'm essentially just noodling how this actually works and therefore I would like to hear any thoughts that people have. Specific examples would be great so I could potentially test them out and learn some Data Science/AI/Machine Learning in the process.

I have no false perception that I could do anything like these quantitative hedge funds. That requires significant capital, use of leverage, hundreds of PH.D data scientist and programmers, costly servers, etc. However, perhaps learning a little bit about their process might improve my fundamental process. Although I'm not even sure about that since perhaps they might be distinctly different. For those wanting to learn more about the investing world, happy to share my knowledge. Also I'm not a data scientist or programmer so while I'd like to learn all these algorithms at the moment this is for my own edification.

Best

Alex

Answer 1

This may not directly answer your questions. There's a class offered by Georgia Tech called Machine Learning for Trading, you might find it useful.

https://www.udacity.com/course/machine-learning-for-trading--ud501

Answer 2

HFT firms are liquidity providers. They post bids and offers at prices around what they believe the fair price of the stock is at the current moment. The distance between those bids and offers can be thought of as a confidence interval. So, to put it quite simply, they can use machine learning to better estimate the fair price of the asset or better estimate the width of their confidence interval.

They are less modeling "human behavior" and more modeling a dynamic system that has random components to it, and they're doing it both quickly and precisely. If you think an asset is currently worth X and you're 80% confident it's worth between X+a and X-b, then if you make a trade at price X+a or X-b you should be pretty happy about it. Obviously its more complex than this, but this should provide you a framework of how you could employ whatever knowledge of machine learning you have towards trading.

Answer 3

Read Max Dama on Automated trading (PDF) - This is the best introduction to algorithmic trading out there: http://www.decal.org/download/2582

Answer 4

Another example might be, if companies that exhibit certain revenue growth metrics, or margin improvement, would that signal a potential buy opportunity? Or perhaps if certain words in their annual report, quarterly filings, press releases indicate this company is likely to do well?

The keys to quantitative investment are research and data analysis.

For example, say you have an idea that companies exhibiting revenue growth will outperform in the future. How do you test that idea? How do you measure the size of the effect (for example, is it large enough to return a profit after trading costs are taken into account?) What are the timescales over which the effect is relevant? How do you ensure that you are measuring a real effect, and not "curve fitting" or "data mining"?

Taking a step back, how do you ensure that your revenue growth data is timely and accurate? How are you measuring outperformance - versus the market as a whole? Versus a sector? Versus a basket of related stocks? Is the effect consistent across sectors, or across countries? Does it hold for both large caps and small caps?

Taking a step forward - once you have identified an effect that you believe in, how do you exploit it? What stocks should you hold in your portfolio, and what residual risks will that expose you to? Are those risks large enough that you should give up some alpha to hedge them? How does the effect interact with other anomalies you have identified?

Quantitative investment firms are not doing anything magical. They are probably looking at a lot of the same data that fundamental investors are looking at (although they have a preference for data that is easy to consume by machines). Indeed, the approach to investment that you said you follow -

Modeling a company's financials, forecasting the cashflows and discounting them back at a discount rate to get the valuation

is essentially a quantitative process. The biggest differences between this and a quantitative investor are

1. A quant investor consumer a greater range of data, and more of it.
2. A quant investor might have an explicit model of the market in mind (described in the language of probability)
3. A quant investor is more systematic, i.e. they try to remove discretion from their investment decisions.

As I said in a comment, I think you have asked a very unfocused question, and as such you will get unfocused answers (like this one). You would do much better to ask a series of shorter, more focused questions if you are really interested in learning more about this area.

Answer 5

You have general and specific questions, so I'll my best here. I have a forex robot that does 30% p.a. 8 years running. It's technical indicators. It's also using one set of rules that is aware of peoples-patterns. (Target prices that traders would commonly sell at). It must be people-aware because even an HFT (and some have failed in big ways) has human ideas and rules in it. I have a Day Trader app that many people use and it has quant-ish things to spot traders habits. I had to build new math indicators (math with logic) to make that work well. I have a LT stock tool I'm just finishing. On 6000 stocks it made an average of 15% per trade - trades once per month based on signal (all technical indicators). I only mention this because I know the road well. 8,000 hours later I'm almost done. The experience summary I'm sharing here is: must be technical and trader-aware. I also have a friend that is a top-hedge fund manager in Switzerland. He used neuroshell (he's not a programmer). That app uses neural network thinking to back-test the best indicators for a given market (any market). However, back-testing isn't a great predictor if done on a single venue (one stock). The forward trades aren't so good (I've done testing intensively for 14 years). Market is driven by belief primarily. The belief (per B.Graham and reality) changes quickly. The time frames you use need different things than the day trader, or swing or investor. There are sentiment indicators of various types for the various time frames. They only help give one "belief" model for that time frame. The best tip I have here is need to look at the bigger timeframes to see where your investment lays. The bigger picture is the best help for trend traders. "Reminiscence of a Stock Operator" is great perspective on trading and markets. Many top traders read that book every year to get grounded in simple truths about markets and people. All the best to you.