# Separating the wheat from the chaff: What quant methods separate skillful managers from lucky ones?

Fund managers are acting in a highly stochastic environment. What methods do you know to systematically separate skillful fund managers from those that were just lucky?

Every idea, reference, paper is welcome! Thank you!

-

Larry Harris has a chapter on performance evaluation in Trading and Exchanges. He states that over a long period of time, a skilled asset manager will consistently have excess returns whereas a lucky one will be expected to have random and unpredictable returns. Thus, we start with the portfolio's market-adjusted return standard deviation:

$$\sigma_{adj} = \sqrt{\sigma^2_{port} + \sigma^2_{mk} - 2\rho\sigma_{port}\sigma_{mk}}$$

where $\rho$ is the correlation between the market and portfolio returns.

For a sample size $n$ (generally number of years), the average excess returns, and the adjusted standard deviation from above, we have a t-statistic:

$$t = \frac{\overline{R_{port}} - \overline{R_{mk}}}{\frac{\sigma_{adj}}{\sqrt{n}}}$$

Now we can simply determine the probability that the manager's excess returns were luck by plugging this t-statistic into the t-distribution's PDF with degrees-of-freedom $n - 1$. The lower the probability, the more we can believe the manager's excess returns were from skill.

-
I feel compelled to point out that plugging your t statistic into the t distribution does not give you the probability that the excess returns were from luck. Instead, it gives you the probability that the fund manager would have earned those returns if he was unskilled (your null hypothesis), and if your distributional assumptions are correct (which they're almost certainly not). –  Chris Taylor Sep 19 '12 at 16:30
@ChrisTaylor When we say that a fund manager is lucky, we mean that he achieved returns without skill; I stand by my original wording. As for whether the t distribution is a valid assumption, it's fine for this question. –  chrisaycock Sep 20 '12 at 3:09
I saw another interesting approach where some researchers (I lost the names) looked for serial correlation in returns within a group. Interestingly they found more serial correlation in negative results than positive results. –  BlueTrin Nov 1 '12 at 14:20
I've purged the comment thread between Freddy and Phil H. This site isn't a discussion forum. –  chrisaycock Feb 20 '13 at 18:52
@chrisaycock: FYI appears there's a popular belief that there maybe issues with your answer, though I personally am not in the position to explain why. –  blunders Sep 1 '13 at 1:27

http://www.scribd.com/doc/34567320/Untangling-Skill-and-Luck

Below is some code that I used recently to illustrate luck (and con-games). The story went like this:

I'll dream up your lucky lottery number for 2010.....let's say it's 20639. The number doesn't matter because we're going to use that for the seed of a random number generator. Then, I'll take the first three digits of your lucky lottery number (206) and reverse them (-.602) and use that as a multiplier on that random number. Since the S&P500 started off in 2010 at about 1100, I'll start the model at that level. Here is the code:

 library(quantmod)

#Read 2010 S&P500 data from Yahoo
tem <- as.zoo(getSymbols("^gspc", from="2010-01-01", to="2011-01-01", auto.assign=FALSE, src = "yahoo"))

#Build a lucky lottery model based on the seed
yt <- tem$GSPC.Adjusted coredata(yt) <- 1100 * exp(-0.602 * cumsum(rnorm(length(yt), sd=0.0113))) #Plot the results plot(tem$GSPC.Adjusted, type="l", main="S&P500 for Year 2010", ylab="S&P500", xlab="", lwd=3, col="darkgray")
lines(yt, lwd=2, col="red")
legend("bottomright", legend=c("Actual S&P500", "Dredged S&P500"), lwd=c(3, 2), col=c("darkgray", "red"))


This is all meaningless BS, but it took a while for some people to untangle luck from the con. The really hard part is to convince yourself that your "skill" actually found something real. Something "real" is a very rare event in the world of investing.

-

In order to have a shot at separating skill from luck, you need a sense of what luck looks like. I think the best chance of understanding luck is to use random portfolios. See, for instance: http://www.portfolioprobe.com/about/random-portfolios-in-finance/

-
+1. Statistical comparisons always require some baseline. As in The Lady Tasting Tea, it's up to you to establish a sensible baseline. One might be able to improve upon completely random baselines by comparing only amongst similar subsets, for example. Or maybe it's better to keep hands off so one doesn't have to justify this or that particular weighting scheme. But then what justifies the uniform weighting scheme? Etc. –  isomorphismes Mar 15 '13 at 22:36
+1 the random portfolio approach offers a good test of the skill of a fund manager. In addition, it is possible to augment the test with the constraints that a manager might be facing, such as asset weights or turnover. –  Felix Sep 29 '14 at 14:27

Read Fooled by Randomness by Nassim Taleb. In a nutshell, he says that you can only tell the difference by understanding the risks that were taken. Lucky investors can win for many years before blowing up. Even if he doesn't blow up, there is no way to know what might have happened if the risks turned out badly.

-

Take a look at White's Reality Check.

Another very crude way would be to calculate a "skill score" (from The Mathematics of Technical Analysis, p325)

$$\tt{skill\ score} = \frac{SKILL\_correct - NOSKILL\_correct}{Total\ decisions - NOSKILL\_correct}$$

• SKILL_correct: the profitable trades
• NOSKILL_correct: randomly assigned trades that were profitable
• Total decisions: number of trades

If this number is 0 or negative, it indicates that you are mostly dealing with a lucky investor, and not a skilled one.

-
Also check out Hansen's test for Superior Predictive Ability, which in most cases is better than White's test. –  Dmitrii I. May 8 '11 at 19:13

I was going to suggest that you use alpha, which is the measure of a managers excess return beyond their benchmark. But here is an alternative view which is quite interesting.

http://money.usnews.com/money/blogs/Fund-Observer/2010/06/24/just-how-lucky-is-your-mutual-fund-manager

-

Check out the last chapter in Grinold's classic Active Portfolio Management (2nd Ed) for a discussion on separating luck from skill

-

I remember an article from graduate school that describes a methodology for measuring the true timing ability of a money manager. I don't remember the name of the article nor the name of the author, however, I do remember some of the details of the article. Maybe someone else has run across it and would be kind enough to post the appropriate reference.

Let's assume that a manager has the ability to be either in cash, earning the risk-free rate, or a long position in a basket of stocks. If the money manager had superior timing ability, he would be in the basket when the basket was returning greater than the risk-free rate, and he would be in a cash position when the basket is returning less than the risk-free rate. What you basically have is a return profile that looks a lot like the payoff of call option.

If you plot market return on the x-axis and manager return on the y-axis, the return should be flat at the risk-free rate for everything to the left of the risk-free rate on the x-axis/ At the risk free-rate on the x-axis, the return should be a 45-degree line up and to the right of the diagram. Over time, you measure manager return against market return, and if he is any good, you should see the call option payoff diagram being roughly drawn out.

-
@user832: Thank you. What was the empirical outcome? –  vonjd May 4 '11 at 5:54

Martijn Cremers and Antti Petajisto have a series of papers using the concept of "Active Share," a new measure of active portfolio management which represents the share of portfolio holdings that differ from the benchmark index holdings, to evaluate mutual fund managers. They find that the most active stock pickers have outperformed their benchmark indices even after fees and transaction costs. In contrast, closet indexers or funds focusing on factor bets have lost to their benchmarks after fees.

Bottom line: when separating skill from luck, concentrate on those managers that actually try to differentiate themselves from the crowd. Also, the more fine grained your strategy is, the more likely it is to represent skill over luck.

-

My 2c worth.

Experience tells me that the better ways to get a feel for whether their strategy is based on something more than luck are amongst:

1) getting to know your traders' -- have a chat, pick their brains, try to get some insight into their methods;

2) see how hard the market has been -- check whether you have just been part of a bull market which basically made pretty-much every strategy a winner, and discount performance accordingly.

I would be skeptical of anything too mathematical for measuring outperformance, since even a badly-designed strategy may have performed well.

I see an analogy with driving: many drivers will successfully get from A to B, but you'll get a much clearer picture of the sorts of risks a driver takes by sitting in the car with them, rather than trying some mathematical analysis of the race. Make sense?

-
Thanks for your \$0.02. Problem with what you're saying is: #1 you can only sit down and chat with so many managers, plus only so many will even be willing to sit down and chat with you. For #2, that doesn't sound like a very sound systematic long-term metric. The point of the question (and the whole site, for that matter) is to get beyond vague generalizations and intuition and use some math to solve these problems. Your answer basically amounts to saying we should give up on trying to answer this question using quant techniques. –  Tal Fishman Jan 30 '12 at 21:05
@TalFishman Both of your points are fair, but your summary of my answer is not. I do not propose that we give up'. Au contraire, I would encourage the quant community to get out there and talk to their traders; we quants generally don't know enough about what traders do (there are questions in quant.stackexchange that express this frustration). Once we get a better understanding, let's put it down in mathematics. –  Robert Jan 31 '12 at 8:44
@Robert: I am with you, +1 on your comment :) –  楊祝昇 Feb 7 '12 at 12:01

A very well thought through exposition on the matter is given in this paper:
A Consultant’s Perspective on Distinguishing Alpha from Noise by John R. Minahan

It combines a lot of wisdom and common sense that sometimes seems to get lost in the process...

-