I want to evaluate a trading strategy. My goal is not to compare it with other strategies, but rather to determine how likely it is that the profits are generated from the strategy itself rather than luck. Indeed, the market volatility is very high and of same order of magnitude than the strategy profits.

My starting point would be to compare the strategy with a large number of random strategies that are backtested for the same time period. By random strategy I mean a strategy where assets are randomly bought and sold. I can then determine the likely-hood that the strategy profits are explainable by luck.

Could you please comment on this test or propose alternative tests?

Many thanks

Edit: The strategy has been running live for one month. Profits refers to live profits, not backtest


Something along these lines is known as the Cowles Test, suggested by Alfred Cowles in 1933 (Can stock market forecasters forecast? https://cowles.yale.edu/sites/default/files/files/pub/misc/cowles-forecasters33.pdf . See page 318 "statistical interpretation of the results."

The strategies in question consisted of getting into the market (go long) at certain times and getting out (sell) at other times. He compiled random records to compare to the record of actual traders as follows:

First determine how many times the trader changed his position (switched from long to flat or vice versa) during the test period.

Then generate as many changes of position as the trader, but at random times during the test period. Evaluate the performance of the random strategy.

The human trader's performance can then be ranked along the artificial traders to see if he performed better than randomly.

Due to lack of suitable random number generators in 1933 Cowles actually used pieces of paper picked out of a hat to generate the random switching times (!). Today with computers the Cowles Test is much easier to perform.


I believe that by "luck" you mean that you want to check if you can attribute the pnl of your strategy to something else than the "alpha" that it's trying to capture.

The standard way of doing this is by using standard market factors (such as Barra's standard risk model for equities say https://www.msci.com/www/research-paper/barra-s-risk-models/014972229 ) and calculate the beta of your portfolio against all those factors. The residual pnl will be what is left of your strategy after you account for those factor returns.

For example the most simple one is to assess how directional your strategy is by calculating the market beta of your portfolio which will tell you how much of your strategy pnl is coming from the overall market return. But there are plenty other such risk factors which you want to account for (momentum, volatility, big cap vs small cap etc...)

Once this is done the natural next step for you is to "control a priori" the exposure to those factors that you want to be orthogonal to. That's one of the core features of the portfolio optimization that you perform in statistical arbitrage: given a set of alphas what is the optimal portfolio that maximizes exposure to them which minimizes transaction costs, balances with risk preference and is orthogonal to a collection of undesired risk factors.

  • 1
    $\begingroup$ Thanks, I heard about these techniques, I will dig into it. $\endgroup$ – David Oct 17 '18 at 12:30
  • $\begingroup$ After extracting the alpha, I still want to evaluate how likely it is that this alpha results from "lucky" trades. I found a satisfying answer by comparing my alpha with the alpha of random strategies $\endgroup$ – David Apr 26 '19 at 9:21
  • $\begingroup$ @David i am not sure what you mean. A random strategy has no alpha. $\endgroup$ – Ezy Apr 26 '19 at 16:32
  • $\begingroup$ In average, no, of course. But a single experiment can have alpha, by chance. If you run a large number of random strategies (same period, same number of trades etc) you can plot the alpha distribution (of course the average alpha is 0). It gives you the probability for each alpha. $\endgroup$ – David Apr 27 '19 at 13:30
  • $\begingroup$ This is exactly what sharpe ratio calculates. It is the t-stat of your strat’s alpha wrt to the null hypothesis of « zero alpha ». So i stil am confused what else you are looking for $\endgroup$ – Ezy Apr 27 '19 at 13:32

To see if a strategy really work or not, you need to run an out of sample backtest. Compare with other random strategy will not work. For example, you can select a period and generate random strategy, there will be always one with relative high sharpe, which beat all the other strategy.

  • $\begingroup$ The strategy has been running live already, it is the live profit I want evaluate. I made this point clear by editing my question. Running a large number of random strategy will of indeed produce some better strategy, but will give an idea of the likely-hood to produce x profit using random strategies. $\endgroup$ – David Oct 17 '18 at 12:12
  • $\begingroup$ If you have out of sample result, then it is easy. Form this to a math problem. Assume your strategy give a profit u on average, with standard sigma, you want to test whether u is significant larger than 0. each time you run your strategy, the results are iid drawn from the distribution given by your strategy. so you need to estimate the u from your experiment and test if it is larger than 0. This is basic hypothesis test problem. $\endgroup$ – XYQ Oct 17 '18 at 23:02
  • $\begingroup$ Thanks! I believe this tells you whether random fluctuations can explain your profit or not. But it doesn't tell you how likely it is that "lucky trades" account for your profit. I found a satisfying answer by comparing my profit to those of random strategies. $\endgroup$ – David Apr 26 '19 at 9:25

There is some research on ranking a strategy against randomly generated positions. See "A uniformly distributed random portfolio" by Kim and Lee 2016

  • $\begingroup$ see marcos lopez deprado's pitfalls of backtesting paper for methods of dealing with the luck factor. don't have time to look for it right now but it's on the net for sure. $\endgroup$ – mark leeds Oct 17 '18 at 4:08
  • $\begingroup$ Thanks for sharing, looks very similar to what I want. $\endgroup$ – David Oct 17 '18 at 12:16
  • 1
    $\begingroup$ @markleeds are you referring to the bias introduced by multiple testing? $\endgroup$ – David Oct 17 '18 at 12:24
  • $\begingroup$ Hi David: I've never actually read the paper but marcos is quite prolific and my take is that it provides a method for of checking whether your profits are a function of the actual strategy or just random. I imagine multiple testing could be discussed but I don't think that's the focus. Sorry for vague answer. $\endgroup$ – mark leeds Oct 17 '18 at 16:24
  • $\begingroup$ Here it is. Sounds like it's more for back-testing but still might be helpful. papers.ssrn.com/sol3/papers.cfm?abstract_id=2326253 $\endgroup$ – mark leeds Oct 17 '18 at 16:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.