Out of Sample Results Decay Rapidly With Prediction Window or Embargo

Question

So I am beginning to dabble my toe into quantitative finance and am trying to validate some model results and am having difficulty thinking about what they tell me.

Here's my situation: I'm trying to forecast returns using an ML model, and am experimenting with different window sizes for my training data, and am logging my n step ahead out-of-sample predictions. I'm also experimenting with an embargo.

So for example if my step is 2, and my embargo is 5, and my window size is 252, then X is 252 training observations, I would make a prediction for the next 2 observations (due to step), and there would be a gap of 5 observations between the training data and out-of-sample predictions.

Here's an abbreviated version of the formula I'm using, in python:

def walk_forward_validation(mod, 
                            X,
                            y,
                            window         = 504, 
                            lookahead      = 1, 
                            step           = 1,
                            embargo        = 0):
    
    """
    Parameters:
      -- mod:  model you are going to use for predictions
      -- X  :  input data
      -- y  :  target variable
      -- window: size of training window you will use
      -- lookahead: how many steps ahead will your model predict
      -- step: how much additional data your model will move ahead after a round of fitting
      -- embargo: how many samples to skip when making out of sample predictions
      """
    
    start_idx   = 0
    stop_idx    = window
    pred_start  = window + embargo
    pred_stop   = pred_start + lookahead
    max_idx     = X.shape[0] - 1
    i = 0
    
    # will store results in these, return as a df at the end
    results = {
        'preds':     [],
        'dates':     [],
        'true_vals': []
    }

    fitting = True
    
    # start the training loop
    while fitting:
        
        # define the training window, fit the model
        X_temp = X.iloc[start_idx:stop_idx]
        y_temp = y.iloc[start_idx:stop_idx]
        
        
        mod.fit(X_temp, y_temp)
        
        # make future predictions, store the results
        X_pred               = X.iloc[pred_start:pred_stop]
        
        # means we've gone past the cutoff point
        if X_pred.shape[0] == 0:
            break
        
        results['preds']     += mod.predict(X_pred).tolist()
        dates                = X_pred.index.values.tolist()
        results['dates']     += dates
        results['true_vals'] += y.iloc[pred_start:pred_stop].values.tolist()
        
        # update the new cutoffs
        start_idx   += step
        stop_idx    += step
        pred_start  += step
        pred_stop   += step
        i += 1
        
        # check to see if we've hit a new cutoff point            
        if stop_idx + lookahead > max_idx:
            offset = (lookahead + stop_idx) - (max_idx - 1)
            lookahead -= offset
    
    results = pd.DataFrame(results)
    results['dates'] = pd.to_datetime(results['dates'])
    
    return pd.DataFrame(results)

What I've done is looked at my model results for varying window sizes, step sizes, and embargo sizes.

Here's a table that summarizes the results:

What's obvious is that as embargo or step get bigger, the results deteriorate quickly. My take on this is that my model is probably highly sensitive to some very temporary patterns in my data that fade quickly, however I'm having a hard time figuring out the following issues:

• How should I choose which set of results to go by? Is there a particular reason one set of validation parameters is more likely to be stable than another? I know about stationary data, and have read Advances in Machine Learning about embargoing, but it's not clear to me that I should automatically choose the validation parameters that gave me the worst results. Let's assume my data doesn't have any look ahead biases.
• If I'm refitting my model on a regular basis to match the timespan of my out-of-sample predictions, is it then alright to go with the more optimistic validation parameters?

For example, let's suppose I'm fitting on hourly data with an embargo of 0 and step of 1. My results have an R2 value of 0.5. Even if these results are temporary in their nature, if I'm also refitting my model every hour, aren't I exploiting spatial correlations rather than being fooled by them?

I understand that results would decay quickly, but if I was unable to continually retrain the model on new data, I'll just stop it, knowing that disaster looms if I don't.

And lastly, is there anything fundamentally wrong with how I'm approaching this issue?

Thanks.

Anyways, sorry for the long post, just trying to get my head in the right place on these issues.

Thank you!

Answer 1

This is a complex question. Let me reformulate its main components to try to give a generic answer:

1. if a relationship is non-stationary and I capture it via a model, I expect the explanatory power of an "outdated" model to be worst that a fresh one
2. once I cross-validated a model, is the "best set of hyper parameters" the one that I can blindly choose?

Start by the second one:

• in machine learning you have to consider 3 sets and not two
  • the learning set
  • the validation set
  • the testing set
• you are meant to choose your hyper parameters on the validation set (that you name embargo) and to testing it on the testing set.
• if the result on the testing set is not good: there is nothing you can do any more or you will face over-fitting.

In your example you somehow suggest to skip the last step, that's probably one of the reasons that you are annoyed.

Now have a look at the non-stationarity aspect: that's true, nevertheless if the efficiency of the model decays very quickly

• you can face an information leakage problem: are you really sure that you did not mixed future observations with past ones?
• you probably should understand what happens: can you document for yourself what exactly means "my model is probably highly sensitive to some very temporary patterns in my data that fade quickly" and convinced that it is something that makes sense?

Once the 2 points are addressed then there is nothing fundamentally wrong in your reasoning.