I'm using a neural network (keras package) to predict Bitcoin prices 48 hours in advance. The issue is that for some reason, my predictions are "correct" but they are lagging behind the true values. I've been struggling with this for weeks. Here is a graph to show you what I mean (true values in red, model predictions in blue):
I think you can see pretty clearly that the overall "shapes" of the two lines match pretty nicely, but the blue one is consistently offset from the red. I know what you're probably thinking: it's the neural network picking up past (autoregressive) values and replicating them into the future because it can't find a better pattern. I CAN ALMOST PROMISE that's not the explanation.
- None of my variables are past/lagged values of the Bitcoin price. As in, ALL variables are exogenous.
- I have tried manually (and by setting shuffle=TRUE) randomizing the order of the training data to remove any possible time-series effect, but the problem persists. I don't have a lot of experience with Neural Networks, but unless there is some other way through which a neural network can replicate past values into the future, I REALLY don't think that's the problem.
I've tried to go through my code to find a place where I've incorrectly set up the tables, but I haven't found an issue yet. Below, please find all my code with explanations. Any help would be enormously appreciated, I can't for the life of me figure out what the problem is.
After importing all of my data and trimming it so that they're all matched by time (first value of BTC price vector is 1/1/2019 2:00, first value of hashrate vector is 1/1/2019 2:00 etc.), this is what I do:
# Actually offsetting the predictors and outcomes
lagg <- 48
# Cutting off first "lagg" outcome entries, so that the "ahead" entries are matched with past predictor entries
bitcoinpricecut <- bitcoinpriceprelag[(lagg):(length(bitcoinpriceprelag))]
The "lagg" variable is just the variable which controls how many hours ahead I'm trying to predict. Interestingly, changing this variable is the ONLY THING that affects the offset. If I make lagg=0, the offset disappears. If I increase it, the offset increases. I cut off the first "lagg" entries of BTC price in order to make the first entry in that vector be a "future" value. Next, I cut off the last "lagg" entries of each predictor variable so that they match the length of BTC price vector
hashrate <- hashrate[1:(length(hashrate)-lagg)]
activeaddresses <- activeaddresses[1:(length(activeaddresses)-lagg)]
difficulty <- difficulty[1:(length(difficulty)-lagg)]
sopr <- sopr[1:(length(sopr)-lagg)]
tethertradingvol <- tethertradingvol[1:(length(tethertradingvol)-lagg)]
tradingvol <- tradingvol[1:(length(tradingvol)-lagg)]
bigaddresseshourly <- bigaddresseshourly[1:(length(bigaddresseshourly)-lagg)]
coindaysdestroyedhourly <- coindaysdestroyedhourly[1:(length(coindaysdestroyedhourly)-lagg)]
exchangeflowhourly <- exchangeflowhourly[1:(length(exchangeflowhourly)-lagg)]
minerrevenuehourly <- minerrevenuehourly[1:(length(minerrevenuehourly)-lagg)]
unrealizedprofitlosshourly <- unrealizedprofitlosshourly[1:(length(unrealizedprofitlosshourly)-lagg)]
tetherrichlisthourly <- tetherrichlisthourly[1:(length(tetherrichlisthourly)-lagg)]
tethersmartcontracthourly <- tethersmartcontracthourly[1:(length(tethersmartcontracthourly)-lagg)]
I then slap all these vectors in a dataframe:
supervised <- data.frame('BitcoinPrice' = bitcoinpricecut)
supervised['HashRate'] <- hashrate
supervised['ActiveAddresses'] <- activeaddresses
supervised['Difficulty'] <- difficulty
supervised['SOPR'] <- sopr
supervised['TetherTradingVol'] <- tethertradingvol
supervised['TradingVol'] <- tradingvol
supervised['AddressesOver10BTC'] <- bigaddresseshourly
supervised['CDD'] <- coindaysdestroyedhourly
supervised['ExchangeNetFlow'] <- exchangeflowhourly
supervised['MinerRevenue'] <- minerrevenuehourly
supervised['UnrealizedProfitLoss'] <- unrealizedprofitlosshourly
supervised['TetherRichList'] <- tetherrichlisthourly
supervised['TetherSmartContracts'] <- tethersmartcontracthourly
Next, I split that dataframe into two, one for training and the other part for testing:
# Splitting into training and testing
N = nrow(supervised)
n = round(N *0.8, digits = 0)
pretrain = supervised[1:(n), ]
pretest = supervised[(n+1):N, ]
I then go ahead and normalize all of values in the training dataset:
recipe_obj <- recipe(BitcoinPrice ~
HashRate
+ ActiveAddresses
+ Difficulty
+ SOPR
+ TetherTradingVol
+ TradingVol
+ AddressesOver10BTC
+ CDD
+ ExchangeNetFlow
+ MinerRevenue
+ UnrealizedProfitLoss
+ TetherRichList
+ TetherSmartContracts,
data=pretrain) %>%
step_normalize(all_predictors()) %>%
step_normalize(all_outcomes()) %>%
prep()
df_processed_tbl <- bake(recipe_obj, pretrain)
Next, I create a dataframe with the same dimensions as the current testing dataframe ("pretest") and fill it with the values from "pretest", but normalized (to normalize these values, I use the mean and standard deviation of the training dataset):
for (testsamp in 1:length(pretest$BitcoinPrice)){
testingdatanorm[testsamp, 'BitcoinPrice'] <- (pretest$BitcoinPrice[testsamp] - recipe_obj$steps[[2]]$means['BitcoinPrice'])/(recipe_obj$steps[[2]]$sds['BitcoinPrice'])
testingdatanorm[testsamp, 'HashRate'] <- (pretest$HashRate[testsamp] - recipe_obj$steps[[1]]$means['HashRate'])/(recipe_obj$steps[[1]]$sds['HashRate'])
testingdatanorm[testsamp, 'ActiveAddresses'] <- (pretest$ActiveAddresses[testsamp] - recipe_obj$steps[[1]]$means['ActiveAddresses'])/(recipe_obj$steps[[1]]$sds['ActiveAddresses'])
testingdatanorm[testsamp, 'Difficulty'] <- (pretest$Difficulty[testsamp] - recipe_obj$steps[[1]]$means['Difficulty'])/(recipe_obj$steps[[1]]$sds['Difficulty'])
testingdatanorm[testsamp, 'SOPR'] <- (pretest$SOPR[testsamp] - recipe_obj$steps[[1]]$means['SOPR'])/(recipe_obj$steps[[1]]$sds['SOPR'])
testingdatanorm[testsamp, 'TetherTradingVol'] <- (pretest$TetherTradingVol[testsamp] - recipe_obj$steps[[1]]$means['TetherTradingVol'])/(recipe_obj$steps[[1]]$sds['TetherTradingVol'])
testingdatanorm[testsamp, 'TradingVol'] <- (pretest$TradingVol[testsamp] - recipe_obj$steps[[1]]$means['TradingVol'])/(recipe_obj$steps[[1]]$sds['TradingVol'])
testingdatanorm[testsamp, 'AddressesOver10BTC'] <- (pretest$AddressesOver10BTC[testsamp] - recipe_obj$steps[[1]]$means['AddressesOver10BTC'])/(recipe_obj$steps[[1]]$sds['AddressesOver10BTC'])
testingdatanorm[testsamp, 'CDD'] <- (pretest$CDD[testsamp] - recipe_obj$steps[[1]]$means['CDD'])/(recipe_obj$steps[[1]]$sds['CDD'])
testingdatanorm[testsamp, 'ExchangeNetFlow'] <- (pretest$ExchangeNetFlow[testsamp] - recipe_obj$steps[[1]]$means['ExchangeNetFlow'])/(recipe_obj$steps[[1]]$sds['ExchangeNetFlow'])
testingdatanorm[testsamp, 'MinerRevenue'] <- (pretest$MinerRevenue[testsamp] - recipe_obj$steps[[1]]$means['MinerRevenue'])/(recipe_obj$steps[[1]]$sds['MinerRevenue'])
testingdatanorm[testsamp, 'UnrealizedProfitLoss'] <- (pretest$UnrealizedProfitLoss[testsamp] - recipe_obj$steps[[1]]$means['UnrealizedProfitLoss'])/(recipe_obj$steps[[1]]$sds['UnrealizedProfitLoss'])
testingdatanorm[testsamp, 'TetherRichList'] <- (pretest$TetherRichList[testsamp] - recipe_obj$steps[[1]]$means['TetherRichList'])/(recipe_obj$steps[[1]]$sds['TetherRichList'])
testingdatanorm[testsamp, 'TetherSmartContracts'] <- (pretest$TetherSmartContracts[testsamp] - recipe_obj$steps[[1]]$means['TetherSmartContracts'])/(recipe_obj$steps[[1]]$sds['TetherSmartContracts'])
}
I then create matrices from the columns (13 predictors) of the pretrain/pretest dataframes in order to use as inputs to my neural network. I'll be honest and say that I don't fully understand the matrix transformations here, I got it from an online tutorial/walkthrough of a NN implementation.
x_train <- df_processed_tbl %>% select(1:13)
x_train <- as.matrix(x_train)
y_train <- df_processed_tbl %>% select(14)
y_train <- as.matrix(y_train)
x_test <- testingdatanorm %>% select(2:14)
x_test <- as.matrix(x_test)
y_test <- testingdatanorm %>% select(1)
y_test <- as.matrix(y_test)
dim(x_train) <- c((length(x_train))/13,1,13)
dim(x_test) <- c((length(x_test))/13,1,13)
length(x_test)
X_shape1 = dim(x_train)[2]
X_shape2 = dim(x_train)[3]
The design of my neural network (I've tried LSTM layers before and it doesn't fix the issue of lag/offset in the prediction). In any case, I doubt there's an issue here, but:
batch_size = 2
model <- keras_model_sequential()
model%>%
layer_dense(units=13,
batch_input_shape = c(batch_size, 1, 13), use_bias = TRUE) %>%
layer_dense(units=75, batch_input_shape = c(batch_size, 1, 13)) %>%
layer_dense(units=1)
model %>% compile(
loss = 'mean_absolute_error',
optimizer = optimizer_adam(lr= 0.00005, decay = 0.00000035),
metrics = c('mean_absolute_error')
)
Here I train the model and create arrays in order to later generate predictions. Again, that's not something I fully understand, but I got it from another NN guide and it seems to work. You'll also notice that I'm only doing 5 Epochs - that's because for some reason, the loss stops decreasing after only 5 Epochs:
Epochs <- 5
for (i in 1:Epochs){
print(i)
model %>% fit(x_train, y_train, epochs=1, batch_size=batch_size, verbose=1, shuffle=FALSE)
}
x_train_arr <- array(data = x_train, dim = c(length(x_train), 1, 1))
y_train_arr <- array(data = y_train, dim = c(length(y_train), 1))
x_test_arr <- array(data = x_test, dim=c(length(x_test),1 ,1))
Finally, after training the model, I generate predictions and reverse the normalization that was originally done:
pred_out <- model %>% predict(x_test, batch_size = batch_size)
pred_out <- as.matrix(pred_out)
norm_history_y <- recipe_obj$steps[[2]]$means['BitcoinPrice']
norm2_history_y <- recipe_obj$steps[[2]]$sds['BitcoinPrice']
nnpredictions <- c()
for (i in 1:length(pred_out)){
nnpredictions <- c(nnpredictions, pred_out[i]*norm2_history_y + norm_history_y)
}
Another dimension tranformation:
dim(nnpredictions) <- c(length(pred_out),1)
Lastly, I apply the reverse-normalization to the "true" values of the testing dataset ("y_test") and prepare everything for ggplot:
y_nntest <- y_test*norm2_history_y
y_nntest <- y_nntest+norm_history_y
y_nntest <- as.data.frame(y_nntest)
nnpredictions <- as.data.frame(nnpredictions)
Using the below code, I generate the graph you saw earlier:
p = ggplot() +
geom_line(data = nnpredictions, aes(x = seq(1, (length(nnpredictions$V1))), y = nnpredictions$V1), color = "blue") +
geom_line(data = y_nntest, aes(x = seq(1, (length(y_nntest$BitcoinPrice))), y = y_nntest$BitcoinPrice), color = "red") +
xlab('Dates') +
ylab('Price')
p
Any help would be appreciated immensely.