# Consistent offset/lag in time-series prediction using Neural Network (all code provided)

I'm using a neural network (keras package) to predict Bitcoin prices 48 hours in advance. The issue is that for some reason, my predictions are "correct" but they are lagging behind the true values. I've been struggling with this for weeks. Here is a graph to show you what I mean (true values in red, model predictions in blue):

I think you can see pretty clearly that the overall "shapes" of the two lines match pretty nicely, but the blue one is consistently offset from the red. I know what you're probably thinking: it's the neural network picking up past (autoregressive) values and replicating them into the future because it can't find a better pattern. I CAN ALMOST PROMISE that's not the explanation.

1. None of my variables are past/lagged values of the Bitcoin price. As in, ALL variables are exogenous.
2. I have tried manually (and by setting shuffle=TRUE) randomizing the order of the training data to remove any possible time-series effect, but the problem persists. I don't have a lot of experience with Neural Networks, but unless there is some other way through which a neural network can replicate past values into the future, I REALLY don't think that's the problem.

I've tried to go through my code to find a place where I've incorrectly set up the tables, but I haven't found an issue yet. Below, please find all my code with explanations. Any help would be enormously appreciated, I can't for the life of me figure out what the problem is.

After importing all of my data and trimming it so that they're all matched by time (first value of BTC price vector is 1/1/2019 2:00, first value of hashrate vector is 1/1/2019 2:00 etc.), this is what I do:

# Actually offsetting the predictors and outcomes
lagg <- 48
# Cutting off first "lagg" outcome entries, so that the "ahead" entries are matched with past predictor entries
bitcoinpricecut <- bitcoinpriceprelag[(lagg):(length(bitcoinpriceprelag))]


The "lagg" variable is just the variable which controls how many hours ahead I'm trying to predict. Interestingly, changing this variable is the ONLY THING that affects the offset. If I make lagg=0, the offset disappears. If I increase it, the offset increases. I cut off the first "lagg" entries of BTC price in order to make the first entry in that vector be a "future" value. Next, I cut off the last "lagg" entries of each predictor variable so that they match the length of BTC price vector

hashrate <- hashrate[1:(length(hashrate)-lagg)]
difficulty <- difficulty[1:(length(difficulty)-lagg)]
sopr <- sopr[1:(length(sopr)-lagg)]
coindaysdestroyedhourly <- coindaysdestroyedhourly[1:(length(coindaysdestroyedhourly)-lagg)]
exchangeflowhourly <- exchangeflowhourly[1:(length(exchangeflowhourly)-lagg)]
minerrevenuehourly <- minerrevenuehourly[1:(length(minerrevenuehourly)-lagg)]
unrealizedprofitlosshourly <- unrealizedprofitlosshourly[1:(length(unrealizedprofitlosshourly)-lagg)]
tetherrichlisthourly <- tetherrichlisthourly[1:(length(tetherrichlisthourly)-lagg)]
tethersmartcontracthourly <- tethersmartcontracthourly[1:(length(tethersmartcontracthourly)-lagg)]


I then slap all these vectors in a dataframe:

supervised <- data.frame('BitcoinPrice' = bitcoinpricecut)
supervised['HashRate'] <- hashrate
supervised['Difficulty'] <- difficulty
supervised['SOPR'] <- sopr
supervised['CDD'] <- coindaysdestroyedhourly
supervised['ExchangeNetFlow'] <- exchangeflowhourly
supervised['MinerRevenue'] <- minerrevenuehourly
supervised['UnrealizedProfitLoss'] <- unrealizedprofitlosshourly
supervised['TetherRichList'] <- tetherrichlisthourly
supervised['TetherSmartContracts'] <- tethersmartcontracthourly


Next, I split that dataframe into two, one for training and the other part for testing:

# Splitting into training and testing
N = nrow(supervised)
n = round(N *0.8, digits = 0)
pretrain = supervised[1:(n), ]
pretest  = supervised[(n+1):N,  ]


I then go ahead and normalize all of values in the training dataset:

recipe_obj <- recipe(BitcoinPrice ~
HashRate
+ Difficulty
+ SOPR
+ CDD
+ ExchangeNetFlow
+ MinerRevenue
+ UnrealizedProfitLoss
+ TetherRichList
+ TetherSmartContracts,
data=pretrain) %>%
step_normalize(all_predictors()) %>%
step_normalize(all_outcomes()) %>%
prep()
df_processed_tbl <- bake(recipe_obj, pretrain)


Next, I create a dataframe with the same dimensions as the current testing dataframe ("pretest") and fill it with the values from "pretest", but normalized (to normalize these values, I use the mean and standard deviation of the training dataset):

for (testsamp in 1:length(pretest$BitcoinPrice)){ testingdatanorm[testsamp, 'BitcoinPrice'] <- (pretest$BitcoinPrice[testsamp] - recipe_obj$steps[[2]]$means['BitcoinPrice'])/(recipe_obj$steps[[2]]$sds['BitcoinPrice'])
testingdatanorm[testsamp, 'HashRate'] <- (pretest$HashRate[testsamp] - recipe_obj$steps[[1]]$means['HashRate'])/(recipe_obj$steps[[1]]$sds['HashRate']) testingdatanorm[testsamp, 'ActiveAddresses'] <- (pretest$ActiveAddresses[testsamp] - recipe_obj$steps[[1]]$means['ActiveAddresses'])/(recipe_obj$steps[[1]]$sds['ActiveAddresses'])
testingdatanorm[testsamp, 'Difficulty'] <- (pretest$Difficulty[testsamp] - recipe_obj$steps[[1]]$means['Difficulty'])/(recipe_obj$steps[[1]]$sds['Difficulty']) testingdatanorm[testsamp, 'SOPR'] <- (pretest$SOPR[testsamp] - recipe_obj$steps[[1]]$means['SOPR'])/(recipe_obj$steps[[1]]$sds['SOPR'])
testingdatanorm[testsamp, 'TetherTradingVol'] <- (pretest$TetherTradingVol[testsamp] - recipe_obj$steps[[1]]$means['TetherTradingVol'])/(recipe_obj$steps[[1]]$sds['TetherTradingVol']) testingdatanorm[testsamp, 'TradingVol'] <- (pretest$TradingVol[testsamp] - recipe_obj$steps[[1]]$means['TradingVol'])/(recipe_obj$steps[[1]]$sds['TradingVol'])
testingdatanorm[testsamp, 'AddressesOver10BTC'] <- (pretest$AddressesOver10BTC[testsamp] - recipe_obj$steps[[1]]$means['AddressesOver10BTC'])/(recipe_obj$steps[[1]]$sds['AddressesOver10BTC']) testingdatanorm[testsamp, 'CDD'] <- (pretest$CDD[testsamp] - recipe_obj$steps[[1]]$means['CDD'])/(recipe_obj$steps[[1]]$sds['CDD'])
testingdatanorm[testsamp, 'ExchangeNetFlow'] <- (pretest$ExchangeNetFlow[testsamp] - recipe_obj$steps[[1]]$means['ExchangeNetFlow'])/(recipe_obj$steps[[1]]$sds['ExchangeNetFlow']) testingdatanorm[testsamp, 'MinerRevenue'] <- (pretest$MinerRevenue[testsamp] - recipe_obj$steps[[1]]$means['MinerRevenue'])/(recipe_obj$steps[[1]]$sds['MinerRevenue'])
testingdatanorm[testsamp, 'UnrealizedProfitLoss'] <- (pretest$UnrealizedProfitLoss[testsamp] - recipe_obj$steps[[1]]$means['UnrealizedProfitLoss'])/(recipe_obj$steps[[1]]$sds['UnrealizedProfitLoss']) testingdatanorm[testsamp, 'TetherRichList'] <- (pretest$TetherRichList[testsamp] - recipe_obj$steps[[1]]$means['TetherRichList'])/(recipe_obj$steps[[1]]$sds['TetherRichList'])
testingdatanorm[testsamp, 'TetherSmartContracts'] <- (pretest$TetherSmartContracts[testsamp] - recipe_obj$steps[[1]]$means['TetherSmartContracts'])/(recipe_obj$steps[[1]]\$sds['TetherSmartContracts'])
}


I then create matrices from the columns (13 predictors) of the pretrain/pretest dataframes in order to use as inputs to my neural network. I'll be honest and say that I don't fully understand the matrix transformations here, I got it from an online tutorial/walkthrough of a NN implementation.

x_train <- df_processed_tbl %>% select(1:13)
x_train <- as.matrix(x_train)
y_train <- df_processed_tbl %>% select(14)
y_train <- as.matrix(y_train)
x_test <- testingdatanorm %>% select(2:14)
x_test <- as.matrix(x_test)
y_test <- testingdatanorm %>% select(1)
y_test <- as.matrix(y_test)
dim(x_train) <- c((length(x_train))/13,1,13)
dim(x_test) <- c((length(x_test))/13,1,13)
length(x_test)
X_shape1 = dim(x_train)[2]
X_shape2 = dim(x_train)[3]


The design of my neural network (I've tried LSTM layers before and it doesn't fix the issue of lag/offset in the prediction). In any case, I doubt there's an issue here, but:

batch_size = 2

model <- keras_model_sequential()
model%>%
layer_dense(units=13,
batch_input_shape = c(batch_size, 1, 13), use_bias = TRUE) %>%
layer_dense(units=75, batch_input_shape = c(batch_size, 1, 13)) %>%
layer_dense(units=1)
model %>% compile(
loss = 'mean_absolute_error',
optimizer = optimizer_adam(lr= 0.00005, decay = 0.00000035),
metrics = c('mean_absolute_error')
)


Here I train the model and create arrays in order to later generate predictions. Again, that's not something I fully understand, but I got it from another NN guide and it seems to work. You'll also notice that I'm only doing 5 Epochs - that's because for some reason, the loss stops decreasing after only 5 Epochs:

Epochs <- 5
for (i in 1:Epochs){
print(i)
model %>% fit(x_train, y_train, epochs=1, batch_size=batch_size, verbose=1, shuffle=FALSE)
}

x_train_arr <- array(data = x_train, dim = c(length(x_train), 1, 1))
y_train_arr <- array(data = y_train, dim = c(length(y_train), 1))
x_test_arr <- array(data = x_test, dim=c(length(x_test),1 ,1))


Finally, after training the model, I generate predictions and reverse the normalization that was originally done:

pred_out <- model %>% predict(x_test, batch_size = batch_size)
pred_out <- as.matrix(pred_out)
norm_history_y <- recipe_obj$$steps[[2]]$$means['BitcoinPrice']
norm2_history_y <- recipe_obj$$steps[[2]]$$sds['BitcoinPrice']
nnpredictions <- c()
for (i in 1:length(pred_out)){
nnpredictions <- c(nnpredictions, pred_out[i]*norm2_history_y + norm_history_y)
}


Another dimension tranformation:

dim(nnpredictions) <- c(length(pred_out),1)


Lastly, I apply the reverse-normalization to the "true" values of the testing dataset ("y_test") and prepare everything for ggplot:

y_nntest <- y_test*norm2_history_y
y_nntest <- y_nntest+norm_history_y
y_nntest <- as.data.frame(y_nntest)
nnpredictions <- as.data.frame(nnpredictions)


Using the below code, I generate the graph you saw earlier:

p = ggplot() +
geom_line(data = nnpredictions, aes(x = seq(1, (length(nnpredictions$$V1))), y = nnpredictions$$V1), color = "blue") +
geom_line(data = y_nntest, aes(x = seq(1, (length(y_nntest$$BitcoinPrice))), y = y_nntest$$BitcoinPrice), color = "red") +
xlab('Dates') +
ylab('Price')
p


Here it is again:

Any help would be appreciated immensely.

• You want to predict returns, not price. Aug 11, 2020 at 21:16
• @Quantoisseur I've heard/read that, but what would that change? Again, there's no time-series structure anyways because I've randomized the training data. (Awesome username btw). Aug 11, 2020 at 21:20
• Without going through your post in detail, this is normally a result of your base price, that the future price prediction is made from, being updated which makes it appear to predict better than it actually is. If you start predicting return labels (maybe positive and negative to start), it will be much easier to evaluate the strategy and produce the backtest. (Thanks) Aug 11, 2020 at 21:31
• @Quantoisseur Every predictor variable I have is exogenous. I am not feeding in any current or past price values. I can try to do this as a classification-type problem like you mention, but I really feel as if that's not the issue. I don't understand how the network can be obtaining any past values to replicate into the future when NONE of the inputs are any price values. Aug 11, 2020 at 21:37
• What is the nature of your exogenous variables? You say they are not lagged values of price but, dependent on the nature of the variables, they may in fact actually be so. For example, if your variables are various technical analysis indicators a sufficiently powerful model may be able to infer past prices and therefore pick up on any autoregressive relationship. Aug 13, 2020 at 10:03

First of all, when you try nonlinear modelling, you should start with a linear model. Did you tried one? what was the result?

It seems that your non linear model for day $$d$$ (blue curve) is close to the value the day before (red curve). You model may be close to $$\hat Y(d)=Y(d-1)+f(Y(d-1),Y(d-2),\ldots;X(d-1),X(d),\ldots),$$ where the non linear part $$f(\cdot)$$ is small. A linear model may be better (or at least not worst).

Moreover (following the discussion in the comments of your question), it is always better to predict stationary variables using stationary variables. Returns are more stationary than prices; that is probably why trying to predict prices ends up with being dominated by the price of yesterday.

• I tried it at some point in the past and it delivered poor results. Tried it again just now and the result is nearly the same as with the neural network above (similar "accuracy", with a very similar lag). Maybe I used other variable with my linear model in the past. This is really odd. Aug 11, 2020 at 23:34
• De Prado claims that there is a hapy medium between price and returns that retains more memory will still being mostly stationary, achieved by Fractional Differentiation... not sure it will help but worth considering. Here is a package that does it: github.com/philipperemy/fractional-differentiation-time-series Aug 12, 2020 at 1:28
• ok @VladimirBelik : so I answered your question (please vote "up" to my answer). This "lag" comes from the non-stationarity of the prices. You should retry on the returns. Aug 20, 2020 at 4:48
• @lehalle Respectfully, that is not the answer to the question. It turned out that the problem was that one of the variables, though it was exogenous, very closely followed current BTC price. As a result, the neural network ended up using that variable as a proxy for past BTC price, which resulted in the replication of past prices into future predictions. Aug 24, 2020 at 16:36
• ok @VladimirBelik, thanks Aug 24, 2020 at 19:27

I might miss a point but if I understand your graph correctly, it seems that the parallelism between the two curves shows that your NN predicts roughly the current price in 2 days (hence the time lag). If you had no algorithm at all but just a prediction that Xt+2 = Xt, you would get a blue curve that exactly replicates the red one with the same lag.

Furthermore, trying to predict future prices is a poor trading model if you keep the loss function unchanged. Indeed, if your error is small but that the foreseen price is above the expected one and that you are long the BTC, you will not complain about that error. Conversely, if the effective price in 2 days is below the predicted one and that you are long BTC short USD, you might loose cash... Therefore, assuming a long only strategy, you should modify your loss function and penalize predictions that are below the effective value during your training.

Hope this helps. Best,

I went through and deleted the variables one-by-one and re-ran the neural network training. What I found was that the quality of the prediction didn't change until I removed a variable called Unrealized Profit/Loss. When I removed it, the prediction became completely different (and very bad). What I realized was that Unrealized Profit/Loss is an exogenous variable, but it very closely follows current BTC price. So, the neural network was falling back on this variable as a proxy for the actual past price, so the past price values were being propagated into future predictions.

Bottom line: Even if all variables are exogenous, if one or more of them closely follow the price, then the variable will be treated as if it WERE the past price. I should have explored my data more thoroughly.