# Testing Statistical Significance of Various Portfolio Simulations

I'm trying to determine which of my portfolio simulations/backtests if any are good enough to put some money into. I outline an approach below and I'm interested in knowing: What problems are there with my approach if any? How big are these problems? Could you recommend a better approach, and explain it with sufficient detail so that I could implement it? - Thanks!!

I ran several portfolio simulations with portfolio sizes ranging from 1 stock to 20 stocks and average holding periods ranging from 2 days to 3 months. The criteria involve various buy/sell/ranking rules and the sims account for all trading costs. Each simulation is run over the prior 10 years, so I have 10 years of the portfolio equity curve. I take the daily simple return (r) of the portfolio equity curve and convert it to the daily lognormal return [LN(1+r)] which I place in column C of a spreadhseet. I want to calculate with 80% confidence what the minimum CAGR of the sim will be 6 months from now. I believe this would be called the 80% VaR. I use the following formula:

80% 6-mo VaR CAGR = [AVERAGE(C:C)-0.69*STDEV(C:C)/SQRT(125)+1]^251-1

I'm interpreting the result as follows: If 80% 6-mo VaR > Benchmark's 10-yr CAGR, then the strategy has an 80% chance of producing alpha 6 months from now and it's worth putting some money into it. I choose 10-year CAGR for the benchmark to match up with the 10-years of data in the backtest. I chose 6-mo VAR because 6-mo is probably the most I'd be willing to wait for confirmation that the strategy is producing alpha. I chose 80% confidence level because I was trying not to be so conservative that I throw away strategies that will most probably produce alpha. After all, having an 80% chance of producing alpha is probably a lot better than the norm. If my purpose was risk measurement, I'm thinking that 99% VaR or 99.9% VaR would be more appropriate.

Note: I used a 0.69 z-score in the above formula, which is different from the normal distribution z-score. In order to account for stock returns having fat tails at the edges, I examined the last 7 years of SPY and IWM daily returns to create my own custom Z-score index that better matches historical LN(1+R) probability distributions. I also examined a highly volatile 1-stock price history as well to account for the fact that portfolios with <20 stocks could have fatter tails than the SPY and IWM. I summarize the custom z-scores I use below:

VaR %-level / Normal Distribution Z-score / Custom Z-Score:

99.9% / 3.09 / 6.25

99% / 2.33 / 3.12

95% / 1.64 / 1.70

80% / 0.84 / 0.69

Thanks!

• "I'm interpreting the result as follows: If 80% 6-mo VaR > Benchmark's 10-yr CAGR, then the strategy has an 80% chance of producing alpha 6 months from now." This would be true if you were testing only one strategy. Instead, you're data mining dozens of strategies, so your results would be biased. – Clark Henry Sep 28 '14 at 2:45
• @nsw What would you recommend then? Would the following fix the data mining problem: Split the data in half, and test in sample on the first 5 years, and out of sample on the last 5 years. Both in and out of sample have to pass the 80% 6-mo VaR > Benchmark's 5-yr CAGR test before putting money into the strategy. – Tarak Sep 28 '14 at 12:43
• Performing out of sample tests is a good idea. But depending on the amount of data mining you're performing, that still might not be sufficient. If you're looking to put money into some strategy you found by data mining, I highly recommend reading a book like David Aronson's "Evidence Based Technical Analysis." – Clark Henry Sep 28 '14 at 16:29