Backtesting Strategies - Sampling and returns types

Question

I am back testing a strategy in R and I have some questions about testing design. I have a universe of around 500 stocks to test filtered based on liquidity. To test the trading strategy I have implemented a simple in-sample/out-of-sample testing scheme. Before running a full test on the whole universe of stocks I have been running the tests over a random sample of the stocks while designing a generic testing framework.

I have been dividing the universe into two parts, I have been optimizing over a random sample of different stocks then using the optimized parameters from the training set to test over a random selection of different stocks from the test set. Are there any problems with doing this rather than testing over the total universe?

My other question is to do with using prices vs returns (vs log returns). I am a little confused as to the reasoning to use returns rather than price to find signals. Would you use returns series because you are more able to compare returns series more easily then price series?

Answer 1

I think the random sampling has some definite advantages. First and foremost, it reduces the risk that you will be "overfitting" by optimizing your strategy's parameters to a certain sample of assets. Second, with such a large asset universe, it would obviously take tons of time and computational resources to test a strategy over historical data of any meaningful length, so you kind of avoid "punishing" yourself for using a long historical timeframe to test on.

That being said, if your end aim is to live trade the strategy on the those 500 assets specifically, maybe you lose something by only optimizing over a random sample of those assets. There's some critical thinking involved here as to how big the random sample should be, how large the parameter space, and so on. So at least part of the answer will depend on your end objectives.

As for using prices vs. signals, IMHO using returns can be really dangerous. Why? When you trade, transactions are done at a given price, not a given return (at least as far as the accounting is concerned). The other thing worth keeping in mind is that a return is dependent on:

(a) What kind of return (simple, geometric, log, etc.)

(b) When the calculation starts (1-year return, 1-day return, etc.)

Unless the strategy is contingent upon the calculation of returns, I would prefer the use of price data instead. (Since computational efficiency appears to be of concern to you given your large dataset, this also has the added advantage of requiring marginally fewer computations since you don't have to calculate returns in order to get your signals.)