Separating the wheat from the chaff: What quant methods separate skillful managers from lucky ones?

Question

Fund managers are acting in a highly stochastic environment. What methods do you know to systematically separate skillful fund managers from those that were just lucky?

Every idea, reference, paper is welcome! Thank you!

Answer 1

Larry Harris has a chapter on performance evaluation in Trading and Exchanges. He states that over a long period of time, a skilled asset manager will consistently have excess returns whereas a lucky one will be expected to have random and unpredictable returns. Thus, we start with the portfolio's market-adjusted return standard deviation:

\begin{equation} \sigma_{adj} = \sqrt{\sigma^2_{port} + \sigma^2_{mk} - 2\rho\sigma_{port}\sigma_{mk}} \end{equation}

where $\rho$ is the correlation between the market and portfolio returns.

For a sample size $n$ (generally number of years), the average excess returns, and the adjusted standard deviation from above, we have a t-statistic:

\begin{equation} t = \frac{\overline{R_{port}} - \overline{R_{mk}}}{\frac{\sigma_{adj}}{\sqrt{n}}} \end{equation}

Now we can simply determine the probability that the manager's excess returns were luck by plugging this t-statistic into the t-distribution's PDF with degrees-of-freedom $n - 1$. The lower the probability, the more we can believe the manager's excess returns were from skill.

Answer 2

Some links:

http://www.scribd.com/doc/34567320/Untangling-Skill-and-Luck

http://web.ist.utl.pt/adriano.simoes/tese/referencias/Papers%20-%20Pedro/UK%20mutual%20fund%20performance%20Skill%20or%20luck.pdf

Below is some code that I used recently to illustrate luck (and con-games). The story went like this:

I'll dream up your lucky lottery number for 2010.....let's say it's 20639. The number doesn't matter because we're going to use that for the seed of a random number generator. Then, I'll take the first three digits of your lucky lottery number (206) and reverse them (-.602) and use that as a multiplier on that random number. Since the S&P500 started off in 2010 at about 1100, I'll start the model at that level. Here is the code:

 library(quantmod)

 #Read 2010 S&P500 data from Yahoo
 tem <- as.zoo(getSymbols("^gspc", from="2010-01-01", to="2011-01-01", auto.assign=FALSE, src = "yahoo"))

 #Build a lucky lottery model based on the seed
 set.seed(20639) #Your lucky lottery number
 yt <- tem$GSPC.Adjusted
 coredata(yt) <- 1100 * exp(-0.602 * cumsum(rnorm(length(yt), sd=0.0113)))

 #Plot the results
 plot(tem$GSPC.Adjusted, type="l", main="S&P500 for Year 2010", ylab="S&P500", xlab="", lwd=3, col="darkgray")
 lines(yt, lwd=2, col="red") 
 legend("bottomright", legend=c("Actual S&P500", "Dredged S&P500"), lwd=c(3, 2), col=c("darkgray", "red"))

This is all meaningless BS, but it took a while for some people to untangle luck from the con. The really hard part is to convince yourself that your "skill" actually found something real. Something "real" is a very rare event in the world of investing.

Answer 3

In order to have a shot at separating skill from luck, you need a sense of what luck looks like. I think the best chance of understanding luck is to use random portfolios. See, for instance: http://www.portfolioprobe.com/about/random-portfolios-in-finance/

Answer 4

Read Fooled by Randomness by Nassim Taleb. In a nutshell, he says that you can only tell the difference by understanding the risks that were taken. Lucky investors can win for many years before blowing up. Even if he doesn't blow up, there is no way to know what might have happened if the risks turned out badly.

Answer 5

Take a look at White's Reality Check.

Another very crude way would be to calculate a "skill score" (from The Mathematics of Technical Analysis, p325)

$$\tt{skill\ score} = \frac{SKILL\_correct - NOSKILL\_correct}{Total\ decisions - NOSKILL\_correct}$$

• SKILL_correct: the profitable trades
• NOSKILL_correct: randomly assigned trades that were profitable
• Total decisions: number of trades

If this number is 0 or negative, it indicates that you are mostly dealing with a lucky investor, and not a skilled one.

Answer 6

I was going to suggest that you use alpha, which is the measure of a managers excess return beyond their benchmark. But here is an alternative view which is quite interesting.

http://money.usnews.com/money/blogs/Fund-Observer/2010/06/24/just-how-lucky-is-your-mutual-fund-manager

Answer 7

Check out the last chapter in Grinold's classic Active Portfolio Management (2nd Ed) for a discussion on separating luck from skill

Answer 8

I remember an article from graduate school that describes a methodology for measuring the true timing ability of a money manager. I don't remember the name of the article nor the name of the author, however, I do remember some of the details of the article. Maybe someone else has run across it and would be kind enough to post the appropriate reference.

Let's assume that a manager has the ability to be either in cash, earning the risk-free rate, or a long position in a basket of stocks. If the money manager had superior timing ability, he would be in the basket when the basket was returning greater than the risk-free rate, and he would be in a cash position when the basket is returning less than the risk-free rate. What you basically have is a return profile that looks a lot like the payoff of call option.

If you plot market return on the x-axis and manager return on the y-axis, the return should be flat at the risk-free rate for everything to the left of the risk-free rate on the x-axis/ At the risk free-rate on the x-axis, the return should be a 45-degree line up and to the right of the diagram. Over time, you measure manager return against market return, and if he is any good, you should see the call option payoff diagram being roughly drawn out.

Answer 9

Martijn Cremers and Antti Petajisto have a series of papers using the concept of "Active Share," a new measure of active portfolio management which represents the share of portfolio holdings that differ from the benchmark index holdings, to evaluate mutual fund managers. They find that the most active stock pickers have outperformed their benchmark indices even after fees and transaction costs. In contrast, closet indexers or funds focusing on factor bets have lost to their benchmarks after fees.

Bottom line: when separating skill from luck, concentrate on those managers that actually try to differentiate themselves from the crowd. Also, the more fine grained your strategy is, the more likely it is to represent skill over luck.

Answer 10

My 2c worth.

Experience tells me that the better ways to get a feel for whether their strategy is based on something more than luck are amongst:

1) `getting to know your traders' -- have a chat, pick their brains, try to get some insight into their methods;

2) see how hard the market has been -- check whether you have just been part of a bull market which basically made pretty-much every strategy a winner, and discount performance accordingly.

I would be skeptical of anything too mathematical for measuring outperformance, since even a badly-designed strategy may have performed well.

I see an analogy with driving: many drivers will successfully get from A to B, but you'll get a much clearer picture of the sorts of risks a driver takes by sitting in the car with them, rather than trying some mathematical analysis of the race. Make sense?

Answer 11

A very well thought through exposition on the matter is given in this paper:
A Consultant’s Perspective on Distinguishing Alpha from Noise by John R. Minahan

It combines a lot of wisdom and common sense that sometimes seems to get lost in the process...