# Out of Sample Results Decay Rapidly With Prediction Window or Embargo

So I am beginning to dabble my toe into quantitative finance and am trying to validate some model results and am having difficulty thinking about what they tell me.

Here's my situation: I'm trying to forecast returns using an ML model, and am experimenting with different window sizes for my training data, and am logging my n step ahead out-of-sample predictions. I'm also experimenting with an embargo.

So for example if my step is 2, and my embargo is 5, and my window size is 252, then X is 252 training observations, I would make a prediction for the next 2 observations (due to step), and there would be a gap of 5 observations between the training data and out-of-sample predictions.

Here's an abbreviated version of the formula I'm using, in python:

def walk_forward_validation(mod,
X,
y,
window         = 504,
step           = 1,
embargo        = 0):

"""
Parameters:
-- mod:  model you are going to use for predictions
-- X  :  input data
-- y  :  target variable
-- window: size of training window you will use
-- embargo: how many samples to skip when making out of sample predictions
"""

start_idx   = 0
stop_idx    = window
pred_start  = window + embargo
max_idx     = X.shape[0] - 1
i = 0

# will store results in these, return as a df at the end
results = {
'preds':     [],
'dates':     [],
'true_vals': []
}

fitting = True

# start the training loop
while fitting:

# define the training window, fit the model
X_temp = X.iloc[start_idx:stop_idx]
y_temp = y.iloc[start_idx:stop_idx]

mod.fit(X_temp, y_temp)

# make future predictions, store the results
X_pred               = X.iloc[pred_start:pred_stop]

# means we've gone past the cutoff point
if X_pred.shape[0] == 0:
break

results['preds']     += mod.predict(X_pred).tolist()
dates                = X_pred.index.values.tolist()
results['dates']     += dates
results['true_vals'] += y.iloc[pred_start:pred_stop].values.tolist()

# update the new cutoffs
start_idx   += step
stop_idx    += step
pred_start  += step
pred_stop   += step
i += 1

# check to see if we've hit a new cutoff point
if stop_idx + lookahead > max_idx:
offset = (lookahead + stop_idx) - (max_idx - 1)

results = pd.DataFrame(results)
results['dates'] = pd.to_datetime(results['dates'])

return pd.DataFrame(results)


What I've done is looked at my model results for varying window sizes, step sizes, and embargo sizes.

Here's a table that summarizes the results:

What's obvious is that as embargo or step get bigger, the results deteriorate quickly. My take on this is that my model is probably highly sensitive to some very temporary patterns in my data that fade quickly, however I'm having a hard time figuring out the following issues:

• How should I choose which set of results to go by? Is there a particular reason one set of validation parameters is more likely to be stable than another? I know about stationary data, and have read Advances in Machine Learning about embargoing, but it's not clear to me that I should automatically choose the validation parameters that gave me the worst results. Let's assume my data doesn't have any look ahead biases.
• If I'm refitting my model on a regular basis to match the timespan of my out-of-sample predictions, is it then alright to go with the more optimistic validation parameters?

For example, let's suppose I'm fitting on hourly data with an embargo of 0 and step of 1. My results have an R2 value of 0.5. Even if these results are temporary in their nature, if I'm also refitting my model every hour, aren't I exploiting spatial correlations rather than being fooled by them?

I understand that results would decay quickly, but if I was unable to continually retrain the model on new data, I'll just stop it, knowing that disaster looms if I don't.

And lastly, is there anything fundamentally wrong with how I'm approaching this issue?

Thanks.

Anyways, sorry for the long post, just trying to get my head in the right place on these issues.

Thank you!

• What exactly is your target variable, if you don't mind me asking? Nov 16, 2021 at 15:45
• return of an index over 6-12 months Nov 16, 2021 at 15:50

This is a complex question. Let me reformulate its main components to try to give a generic answer:

1. if a relationship is non-stationary and I capture it via a model, I expect the explanatory power of an "outdated" model to be worst that a fresh one
2. once I cross-validated a model, is the "best set of hyper parameters" the one that I can blindly choose?

Start by the second one:

• in machine learning you have to consider 3 sets and not two
• the learning set
• the validation set
• the testing set
• you are meant to choose your hyper parameters on the validation set (that you name embargo) and to testing it on the testing set.
• if the result on the testing set is not good: there is nothing you can do any more or you will face over-fitting.

In your example you somehow suggest to skip the last step, that's probably one of the reasons that you are annoyed.

Now have a look at the non-stationarity aspect: that's true, nevertheless if the efficiency of the model decays very quickly

• you can face an information leakage problem: are you really sure that you did not mixed future observations with past ones?
• you probably should understand what happens: can you document for yourself what exactly means "my model is probably highly sensitive to some very temporary patterns in my data that fade quickly" and convinced that it is something that makes sense?

Once the 2 points are addressed then there is nothing fundamentally wrong in your reasoning.

• A point about cross validation: I'm not changing my model parameters, just the length of time I test my predictions over. So I'm not really trying evaluate what a change in the model would do. Nov 16, 2021 at 14:30
• I considered that the "step" is an hyperparameter, but you are right: I tried to answer a very generic way and I hope that it nevertheless corresponds to your expectation. Nov 16, 2021 at 18:44