1
$\begingroup$

Goal: A team and I are looking to build a model that performs a predictive action for the state of the market on day T + n, using the data at hand on day T. To build this model, I'm using an EOD market data source going back until the early 1990s.

Moreover, we are looking to separate the dataset into training and testing subsets, in order to optimize a model parameter, then to eventually test its performance in the out-of-sample subset.

Problem: The initial attempt is to split the dataset by calendar year, and randomly assign each year into either the testing or training set. However, the following issues have been voiced:

  • Since our model is attempting to predict n days into the future (we can assume n is less than, say 15 or 20, but likely greater than 2-3), our training dataset needs to pull the market state from n days in the future in order to do our analysis. This would seem to indicate the we either need to pull n days from the testing dataset, or that we would need to drop the last n days from training set
  • Any given n-day span during the year may have some significance for our analysis, whether it's an earnings report, or otherwise. In particular, the n-day window that ends the calendar year is considered an important period for our analysis, so dropping these n border points is not ideal (and also might result in systematic model bias)

Question: Is there a proper way to partition training and testing datasets, given that our analysis requires that we use a the datapoint that occurs n days in the future?

$\endgroup$
  • $\begingroup$ Why do you need to randomly assign years to buckets instead of split it right down the middle and capture market cycles? Also, why calendar years? If calendar year end points are so important, just choose another random 365 day time span and accept that the last 3 days can’t be tested. Say something like this to yourself: “Granny gave me $50 for my birthday. How much sweet mullah would I have to spend on my next birthday if she goes broke and I gotta rely on my trading strategy?” $\endgroup$ – Mild_Thornberry Nov 26 '19 at 0:31
  • $\begingroup$ Hi @Mild_Thornberry, thanks for the comment! The motivation for shuffling the training and testing years is to try to make the two samples similar in nature to avoid any model bias. As mentioned, we are looking to solve for a parameter using the training dataset, and test how that parameter does on the testing dataset. Thanks again for your comment. Also, you will need to explain the Granny quote, as I am not quite sure what you've meant by that! $\endgroup$ – David C Nov 26 '19 at 16:27
  • $\begingroup$ Can you split your dataset into two full economic cycles? 1997-2003 and 2003-2009? That’d avoid model bias. My granny example was just a goofy way to say: if your investment strategy works, it should work no matter when you start it. You can begin your strategy on your birthday, on 1/1, or on any random day in between. If the end of the year is important, choose a different range of days, like 6/30-6/30, or 2/14-2/14, or whatever! $\endgroup$ – Mild_Thornberry Nov 30 '19 at 18:42
0
$\begingroup$

For situations like this, I've typically considered the dataset as a whole and simply plotted, or in some other way evaluated, a model estimate relative to an actual value over a rolling period. For instance in the case volatility modeling, compare model-estimated (via ARCH/GARCH, implied vol, etc) vol to realized vol on a rolling basis, making sure the calculation periods coincide.

Insofar as your model is dynamic, it doesn't make sense to fit the model on data from 1996 and then test it on data from 2007, unless you expect it to work.

It's also going to be problematic to approach it as you describe given the +/-n day look ahead/back around beginning and end of period (year).

Regarding your second bullet, I'm not clear why you'd drop n border points. The issue you mention about earnings announcements falling within your n-day window will likely be a limitation of your model though unless you make some kind of accommodation. For instance, to take it back to vol modeling again, you see spikes in vol at both the beginning and end of days and the beginning and end of weeks. It's easy enough to get around this by considering something akin to a seasonality adjustment or to simply look over long enough periods that the spikes even out.

You'd probably want to make some other accommodation to deal with this in your case (eg, earnings announcements are public knowledge well beforehand, make some adjustment to your model to work around them).

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.