1
$\begingroup$

As an amateur trader, I have reasonable success with my current end-of-day trading systems.

All my systems are based on at least ONE year of "in-sample" data for backtesting, followed by 6-months of "out-of-sample" data.

So my question is: Could I take TWO years of "in-sample" data, but ignore any data for periods in which I would not have traded due to the market conditions?

(I used the "data" tag, and searched back to November 2011. I searched all posts for "Historical data".I could not find any related posts.)

Example: Assume I want to trade a "long-only" system, in an "upward-trending" market.

Assume the chart for the previous two years looked like a "W". The first 6 months trends down, then 6 months up, then down, then up.

Can I simply ignore the first and third 6-month periods, and use only the second and fourth 6-month periods?

Reason: If the market was trending down, I would not trade. So, why should my system be based on any previous market data that was trending downwards?

Method:

  • In the "W" example above, copy and paste the second 6-month data into Excel.
  • Ignore the third 6-month data.
  • Copy and paste the fourth 6-month data into Excel.
  • But adjust the entire second batch of data upwards so that it joins seamlessly to the end of the first batch. (Possibly by making the first price of the second batch equal to the last price of the first batch.)
  • So, the result would be a 1-year data series that only trended upwards.

I would be most grateful for any comments.

(Added after Nathan S's answer below. My addition is too long to be added as a comment to Nathan S's answer.)

It all depends to me on whether your system can categorize a price series as "upward trending."

My first impression was that this is a valid point. So I was about to defend my trend indicator (TE). But, on reflection, I now think it's a distraction. It occurred to me that people such as engineers, IT troubleshooters, and people who conduct human trials always attempt to isolate the "common factor(s)" between two sets of experiments. After that, any differences between the results of the experiments will depend only on differences between the components.

In my case, I think the TE (good or bad) is a common factor. It has already been optimized on a watchlist using 3 years of "undoctored" market data. I use that TE for any system based on that watchlist. (For other watchlists, I have different TEs.)

So the TE will be common to both sets of experiments (doctored data versus undoctored data). For the doctored data, obviously I need to extend the starting point further back in time in order to capture the required number of bars (days) of doctored data.

Here are some statements that I have seen regarding the "selection" or "cherrypicking" of data for backtesting. Unfortunately, I have no links, because I tended to agree with all these statements. (I'm more likely to keep links for statements that I disagree with, for later research.)

Don't assume that a system that works with one of the following will work with the other:

  • US stocks versus Australian stocks.
  • A portfolio of Financials versus a portfolio of Utilities.
  • A bull market versus a bear market.

I think it was the final statement that made me "invent" my own statement: Don't optimize an "uptrend" system using down-trending data.

Improve this question
$\endgroup$
1
  • $\begingroup$ I think you're okay as long as you ignore the periods via your model. Don't just skip the dates manually. Your model will need to live in the real world so it will need to know how to identify and respond to conditions. $\endgroup$
    – Eric
    Commented Mar 9, 2015 at 13:34

2 Answers 2

Reset to default
1
$\begingroup$

It all depends to me on whether your system can categorize a price series as "upward trending." I want objective, programmed rules. If my system includes a market condition filter then not only can I ignore cases that would not pass it, but I must ignore them to measure the system.

If I invent the rule to optimize performance that ran cold in some tested periods, (Not saying that's how your market conditions filter was concocted, but I have done this) then I'm very wary and even more skeptical with out of sample.

Improve this answer
$\endgroup$
1
$\begingroup$

You have to be careful when saying "I would have done this". It's too easy in backtesting to make this mistake. From your description of the data, you have no way of knowing it was in a downtrend, until the downtrend was over or ,at least already in full swing. Nathan S's answer and radpin's comments are exactly what you have to do.

I didn't answer just to tell you to listen to other answers but to give an example of things I do. I know that if a market closes at noon one day, I wouldn't have traded and I would know that in advance. I can also make a volume cutoff so I don't trade over the xmas holidays for example. The system only knows this after the fact but I can safely assume I know when the holidays are, so I ignore discard those results after the fact.

Improve this answer
$\endgroup$
2
  • 1
    $\begingroup$ Thanks to all three responders. I'm convinced that all the answers are identical to the answers that I might have given myself, if someone had asked me the same question 2 years ago. But I now have this annoying, persistent nag at the back of my mind. It's probably just another of my "senior moments". I think the only way to get rid of the nag is to do the two tests with two sets of data - "real" and "doctored". The only thing that's stopping me is the amount of time I might take to doctor the data. I think there are many "non-trading" examples of using an "engineered" test environment..... $\endgroup$
    – OldGrantonian
    Commented Mar 11, 2015 at 13:57
  • $\begingroup$ ....to match the future intended operational conditions. For example, family car manufacturers don't test their vehicles over bumpy terrain, through rain-filled ditches, and over sand dunes in pitch blackness. The tank manufacturer does not test his vehicles on smooth, level roads in perfect ambient conditions. $\endgroup$
    – OldGrantonian
    Commented Mar 11, 2015 at 13:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.