Consistent offset/lag in time-series prediction using Neural Network (all code provided)

Question

I'm using a neural network (keras package) to predict Bitcoin prices 48 hours in advance. The issue is that for some reason, my predictions are "correct" but they are lagging behind the true values. I've been struggling with this for weeks. Here is a graph to show you what I mean (true values in red, model predictions in blue):

I think you can see pretty clearly that the overall "shapes" of the two lines match pretty nicely, but the blue one is consistently offset from the red. I know what you're probably thinking: it's the neural network picking up past (autoregressive) values and replicating them into the future because it can't find a better pattern. I CAN ALMOST PROMISE that's not the explanation.

1. None of my variables are past/lagged values of the Bitcoin price. As in, ALL variables are exogenous.
2. I have tried manually (and by setting shuffle=TRUE) randomizing the order of the training data to remove any possible time-series effect, but the problem persists. I don't have a lot of experience with Neural Networks, but unless there is some other way through which a neural network can replicate past values into the future, I REALLY don't think that's the problem.

I've tried to go through my code to find a place where I've incorrectly set up the tables, but I haven't found an issue yet. Below, please find all my code with explanations. Any help would be enormously appreciated, I can't for the life of me figure out what the problem is.

After importing all of my data and trimming it so that they're all matched by time (first value of BTC price vector is 1/1/2019 2:00, first value of hashrate vector is 1/1/2019 2:00 etc.), this is what I do:

# Actually offsetting the predictors and outcomes
lagg <- 48
# Cutting off first "lagg" outcome entries, so that the "ahead" entries are matched with past predictor entries
bitcoinpricecut <- bitcoinpriceprelag[(lagg):(length(bitcoinpriceprelag))]

The "lagg" variable is just the variable which controls how many hours ahead I'm trying to predict. Interestingly, changing this variable is the ONLY THING that affects the offset. If I make lagg=0, the offset disappears. If I increase it, the offset increases. I cut off the first "lagg" entries of BTC price in order to make the first entry in that vector be a "future" value. Next, I cut off the last "lagg" entries of each predictor variable so that they match the length of BTC price vector

hashrate <- hashrate[1:(length(hashrate)-lagg)]
activeaddresses <- activeaddresses[1:(length(activeaddresses)-lagg)]
difficulty <- difficulty[1:(length(difficulty)-lagg)]
sopr <- sopr[1:(length(sopr)-lagg)]
tethertradingvol <- tethertradingvol[1:(length(tethertradingvol)-lagg)]
tradingvol <- tradingvol[1:(length(tradingvol)-lagg)]
bigaddresseshourly <- bigaddresseshourly[1:(length(bigaddresseshourly)-lagg)]
coindaysdestroyedhourly <- coindaysdestroyedhourly[1:(length(coindaysdestroyedhourly)-lagg)]
exchangeflowhourly <- exchangeflowhourly[1:(length(exchangeflowhourly)-lagg)]
minerrevenuehourly <- minerrevenuehourly[1:(length(minerrevenuehourly)-lagg)]
unrealizedprofitlosshourly <- unrealizedprofitlosshourly[1:(length(unrealizedprofitlosshourly)-lagg)]
tetherrichlisthourly <- tetherrichlisthourly[1:(length(tetherrichlisthourly)-lagg)]
tethersmartcontracthourly <- tethersmartcontracthourly[1:(length(tethersmartcontracthourly)-lagg)]

I then slap all these vectors in a dataframe:

supervised <- data.frame('BitcoinPrice' = bitcoinpricecut)
supervised['HashRate'] <- hashrate 
supervised['ActiveAddresses'] <- activeaddresses 
supervised['Difficulty'] <- difficulty 
supervised['SOPR'] <- sopr
supervised['TetherTradingVol'] <- tethertradingvol 
supervised['TradingVol'] <- tradingvol 
supervised['AddressesOver10BTC'] <- bigaddresseshourly 
supervised['CDD'] <- coindaysdestroyedhourly 
supervised['ExchangeNetFlow'] <- exchangeflowhourly 
supervised['MinerRevenue'] <- minerrevenuehourly 
supervised['UnrealizedProfitLoss'] <- unrealizedprofitlosshourly 
supervised['TetherRichList'] <- tetherrichlisthourly 
supervised['TetherSmartContracts'] <- tethersmartcontracthourly 

Next, I split that dataframe into two, one for training and the other part for testing:

# Splitting into training and testing
N = nrow(supervised)
n = round(N *0.8, digits = 0)
pretrain = supervised[1:(n), ]
pretest  = supervised[(n+1):N,  ]

I then go ahead and normalize all of values in the training dataset:

recipe_obj <- recipe(BitcoinPrice ~ 
                      HashRate 
                     + ActiveAddresses 
                     + Difficulty 
                     + SOPR 
                     + TetherTradingVol 
                     + TradingVol 
                     + AddressesOver10BTC 
                     + CDD 
                     + ExchangeNetFlow 
                     + MinerRevenue 
                     + UnrealizedProfitLoss 
                     + TetherRichList 
                     + TetherSmartContracts, 
                     data=pretrain) %>%
              step_normalize(all_predictors()) %>%
              step_normalize(all_outcomes()) %>%
              prep()
df_processed_tbl <- bake(recipe_obj, pretrain)

Next, I create a dataframe with the same dimensions as the current testing dataframe ("pretest") and fill it with the values from "pretest", but normalized (to normalize these values, I use the mean and standard deviation of the training dataset):

for (testsamp in 1:length(pretest$BitcoinPrice)){
  testingdatanorm[testsamp, 'BitcoinPrice'] <- (pretest$BitcoinPrice[testsamp] - recipe_obj$steps[[2]]$means['BitcoinPrice'])/(recipe_obj$steps[[2]]$sds['BitcoinPrice'])
  testingdatanorm[testsamp, 'HashRate'] <- (pretest$HashRate[testsamp] - recipe_obj$steps[[1]]$means['HashRate'])/(recipe_obj$steps[[1]]$sds['HashRate'])
  testingdatanorm[testsamp, 'ActiveAddresses'] <- (pretest$ActiveAddresses[testsamp] - recipe_obj$steps[[1]]$means['ActiveAddresses'])/(recipe_obj$steps[[1]]$sds['ActiveAddresses'])
  testingdatanorm[testsamp, 'Difficulty'] <- (pretest$Difficulty[testsamp] - recipe_obj$steps[[1]]$means['Difficulty'])/(recipe_obj$steps[[1]]$sds['Difficulty'])
  testingdatanorm[testsamp, 'SOPR'] <- (pretest$SOPR[testsamp] - recipe_obj$steps[[1]]$means['SOPR'])/(recipe_obj$steps[[1]]$sds['SOPR'])
  testingdatanorm[testsamp, 'TetherTradingVol'] <- (pretest$TetherTradingVol[testsamp] - recipe_obj$steps[[1]]$means['TetherTradingVol'])/(recipe_obj$steps[[1]]$sds['TetherTradingVol'])
  testingdatanorm[testsamp, 'TradingVol'] <- (pretest$TradingVol[testsamp] - recipe_obj$steps[[1]]$means['TradingVol'])/(recipe_obj$steps[[1]]$sds['TradingVol'])
  testingdatanorm[testsamp, 'AddressesOver10BTC'] <- (pretest$AddressesOver10BTC[testsamp] - recipe_obj$steps[[1]]$means['AddressesOver10BTC'])/(recipe_obj$steps[[1]]$sds['AddressesOver10BTC'])
  testingdatanorm[testsamp, 'CDD'] <- (pretest$CDD[testsamp] - recipe_obj$steps[[1]]$means['CDD'])/(recipe_obj$steps[[1]]$sds['CDD'])
  testingdatanorm[testsamp, 'ExchangeNetFlow'] <- (pretest$ExchangeNetFlow[testsamp] - recipe_obj$steps[[1]]$means['ExchangeNetFlow'])/(recipe_obj$steps[[1]]$sds['ExchangeNetFlow'])
  testingdatanorm[testsamp, 'MinerRevenue'] <- (pretest$MinerRevenue[testsamp] - recipe_obj$steps[[1]]$means['MinerRevenue'])/(recipe_obj$steps[[1]]$sds['MinerRevenue'])
  testingdatanorm[testsamp, 'UnrealizedProfitLoss'] <- (pretest$UnrealizedProfitLoss[testsamp] - recipe_obj$steps[[1]]$means['UnrealizedProfitLoss'])/(recipe_obj$steps[[1]]$sds['UnrealizedProfitLoss'])
  testingdatanorm[testsamp, 'TetherRichList'] <- (pretest$TetherRichList[testsamp] - recipe_obj$steps[[1]]$means['TetherRichList'])/(recipe_obj$steps[[1]]$sds['TetherRichList'])
  testingdatanorm[testsamp, 'TetherSmartContracts'] <- (pretest$TetherSmartContracts[testsamp] - recipe_obj$steps[[1]]$means['TetherSmartContracts'])/(recipe_obj$steps[[1]]$sds['TetherSmartContracts'])
  }

I then create matrices from the columns (13 predictors) of the pretrain/pretest dataframes in order to use as inputs to my neural network. I'll be honest and say that I don't fully understand the matrix transformations here, I got it from an online tutorial/walkthrough of a NN implementation.

x_train <- df_processed_tbl %>% select(1:13)
x_train <- as.matrix(x_train)
y_train <- df_processed_tbl %>% select(14)
y_train <- as.matrix(y_train)
x_test <- testingdatanorm %>% select(2:14)
x_test <- as.matrix(x_test)
y_test <- testingdatanorm %>% select(1)
y_test <- as.matrix(y_test)
dim(x_train) <- c((length(x_train))/13,1,13)
dim(x_test) <- c((length(x_test))/13,1,13)
length(x_test)
X_shape1 = dim(x_train)[2]
X_shape2 = dim(x_train)[3]

The design of my neural network (I've tried LSTM layers before and it doesn't fix the issue of lag/offset in the prediction). In any case, I doubt there's an issue here, but:

batch_size = 2          

model <- keras_model_sequential()
model%>%
  layer_dense(units=13, 
             batch_input_shape = c(batch_size, 1, 13), use_bias = TRUE) %>%
  layer_dense(units=75, batch_input_shape = c(batch_size, 1, 13)) %>%
  layer_dense(units=1)
model %>% compile(
  loss = 'mean_absolute_error',
  optimizer = optimizer_adam(lr= 0.00005, decay = 0.00000035),  
  metrics = c('mean_absolute_error')
)

Here I train the model and create arrays in order to later generate predictions. Again, that's not something I fully understand, but I got it from another NN guide and it seems to work. You'll also notice that I'm only doing 5 Epochs - that's because for some reason, the loss stops decreasing after only 5 Epochs:

Epochs <- 5
for (i in 1:Epochs){
  print(i)
  model %>% fit(x_train, y_train, epochs=1, batch_size=batch_size, verbose=1, shuffle=FALSE)
}

x_train_arr <- array(data = x_train, dim = c(length(x_train), 1, 1))
y_train_arr <- array(data = y_train, dim = c(length(y_train), 1))
x_test_arr <- array(data = x_test, dim=c(length(x_test),1 ,1))

Finally, after training the model, I generate predictions and reverse the normalization that was originally done:

pred_out <- model %>% predict(x_test, batch_size = batch_size) 
pred_out <- as.matrix(pred_out)
norm_history_y <- recipe_obj$steps[[2]]$means['BitcoinPrice']
norm2_history_y <- recipe_obj$steps[[2]]$sds['BitcoinPrice']
nnpredictions <- c()
for (i in 1:length(pred_out)){
  nnpredictions <- c(nnpredictions, pred_out[i]*norm2_history_y + norm_history_y)
}

Another dimension tranformation:

dim(nnpredictions) <- c(length(pred_out),1)

Lastly, I apply the reverse-normalization to the "true" values of the testing dataset ("y_test") and prepare everything for ggplot:

y_nntest <- y_test*norm2_history_y
y_nntest <- y_nntest+norm_history_y
y_nntest <- as.data.frame(y_nntest)
nnpredictions <- as.data.frame(nnpredictions)

Using the below code, I generate the graph you saw earlier:

p = ggplot() + 
  geom_line(data = nnpredictions, aes(x = seq(1, (length(nnpredictions$V1))), y = nnpredictions$V1), color = "blue") +
  geom_line(data = y_nntest, aes(x = seq(1, (length(y_nntest$BitcoinPrice))), y = y_nntest$BitcoinPrice), color = "red") +
  xlab('Dates') +
  ylab('Price')
p

Here it is again:

Any help would be appreciated immensely.

Answer 1

First of all, when you try nonlinear modelling, you should start with a linear model. Did you tried one? what was the result?

It seems that your non linear model for day $d$ (blue curve) is close to the value the day before (red curve). You model may be close to $$\hat Y(d)=Y(d-1)+f(Y(d-1),Y(d-2),\ldots;X(d-1),X(d),\ldots),$$ where the non linear part $f(\cdot)$ is small. A linear model may be better (or at least not worst).

Moreover (following the discussion in the comments of your question), it is always better to predict stationary variables using stationary variables. Returns are more stationary than prices; that is probably why trying to predict prices ends up with being dominated by the price of yesterday.

Answer 2

I went through and deleted the variables one-by-one and re-ran the neural network training. What I found was that the quality of the prediction didn't change until I removed a variable called Unrealized Profit/Loss. When I removed it, the prediction became completely different (and very bad). What I realized was that Unrealized Profit/Loss is an exogenous variable, but it very closely follows current BTC price. So, the neural network was falling back on this variable as a proxy for the actual past price, so the past price values were being propagated into future predictions.

Bottom line: Even if all variables are exogenous, if one or more of them closely follow the price, then the variable will be treated as if it WERE the past price. I should have explored my data more thoroughly.

Answer 3

I might miss a point but if I understand your graph correctly, it seems that the parallelism between the two curves shows that your NN predicts roughly the current price in 2 days (hence the time lag). If you had no algorithm at all but just a prediction that Xt+2 = Xt, you would get a blue curve that exactly replicates the red one with the same lag.

Furthermore, trying to predict future prices is a poor trading model if you keep the loss function unchanged. Indeed, if your error is small but that the foreseen price is above the expected one and that you are long the BTC, you will not complain about that error. Conversely, if the effective price in 2 days is below the predicted one and that you are long BTC short USD, you might loose cash... Therefore, assuming a long only strategy, you should modify your loss function and penalize predictions that are below the effective value during your training.

Hope this helps. Best,