0
$\begingroup$

Aims:

  • Given approximately 11 years of historical time-series data, to determine how much of this data should be reserved for in-sample and out-of-sample testing when using the data in the backtesting of a forecasting system
  • To become familiar with potential techniques, if any, that can be used to determine an in-sample and out-of-sample split to reduce the likelihood of the over-fitting modelling error

Details:

As observable from the chart seen below, the time-series data spans from March 2012 to October 2023. During this approximate 11.5 year period, various market states/ conditions can be seen. For example:

  • 2012 to 2014: up-trending market
  • 2014 to 2016: down-trending market
  • 2016 to 2018: sideways-moving market
  • 2019 onward: substantial up-trend and highly volatile, then steep down-trend

Additionally, from the end of 2019, the data was likely sensitive to real-world inputs including the effects of the COVID-19 pandemic which were not present earlier on in the sample.

enter image description here

Potential Approaches:

  • A simple 50:50 in-sample to out-of-sample split - the first half for testing and optimising, the second half for evaluating performance
  • A 70:30 split so the system is built around the majority of the data before moving on to performance evaluation (this could be more susceptible to over-fitting)
  • Given the various market conditions listed above, using a 50:50 split on each period i.e. the bullish market during 2012-2014 would be divided so the first half was for in-sample and the second half for out-of-sample. This process would be repeated for the bearish market from 2014-2016, the sideways market for 2016-2018, and the highly volatile and bullish market 2019 onward

Questions:

Q1. Based on the characteristics of the time-series data, are there any ways that this data could be divided or would a 50:50 split be sufficient?

Q2. Are there any techniques that would help in determining an appropriate sample split, or is this very much a discretionary process that is unique to each sample set?

$\endgroup$

0

This site is temporarily in read-only mode and not accepting new answers.