2
$\begingroup$

Please bear with me through the whole question - I just want to make it very clear what I've done so far and why I'm so perplexed.

I am working with a neural network with the Keras package in R, trying to predict hourly Bitcoin price, 24 hours ahead. Here is the code for my model:

batch_size = 2              

model <- keras_model_sequential()
model%>%
  layer_dense(units=13, 
             batch_input_shape = c(batch_size, 1, 13), use_bias = TRUE) %>%
  layer_dense(units=17, batch_input_shape = c(batch_size, 1, 13)) %>%
  layer_dense(units=1)
model %>% compile(
  loss = 'mean_squared_error',
  optimizer = optimizer_adam(lr= 0.000025, decay = 0.0000015),  
  metrics = c('mean_squared_error')
)
summary(model)

Epochs <- 25
for (i in 1:Epochs){
  print(i)
  model %>% fit(x_train, y_train, epochs=1, batch_size=batch_size, verbose=1, shuffle=TRUE)
  #model %>% reset_states()
}

You may notice that I am working on a time-series problem, but not using LSTM. This is because none of my inputs are time-series values. They are all exogenous variables. You'll also notice that I commented out the line "model %>% reset_states()". I'm not sure if that's the right thing to do here, but from what I read, that line is for LSTM models and since I'm not using one anymore, I commented it out.

Again, because I have no time-series inputs, I also set "shuffle=" to TRUE. So, below are the predictions in blue vs. true value in red: First Pic

You can see that the predictions very often (but not always) lag behind the true value. Additionally, this lag is not constant. Let me again emphasize that all of the variables are exogenous. There is not past-price input that the model could be using in order to generate these late predictions. AND don't forget I set shuffle= TRUE, which confuses me even further as to how the model could be giving such results if there's no way (that I know of) that it can "see" past values in order to replicate them. Here is the graph of the training data fit: Training Data fit

It's harder to tell, but the lag exists in the training data as well. I'll also say that if I change how far ahead I'm trying to predict, the apparent prediction lag changes too. If I try to predict 0 hours ahead (so I'm "predicting" current price given current conditions), there is no lag in prediction.

I checked to make sure the tables/columns were set up right so that the model is trained on "current" conditions predicting price 24 hours ahead. I've also played around with the network architecture and batch size. The only thing that seems to affect this lag is how far ahead I'm trying to predict - that's to say, how many rows I shifted my Bitcoin Price column by so that "future" price rows are matched with past predictor rows.

Something else weird I've noticed is that with this non-LSTM model, it has a rather high error when training (MSE=0.07) which it reaches after only 5-9 Epochs and then doesn't go any lower. I don't think this is relevant because the LSTM model I used before achieved MSE=0.005 and still had the same lag issue, but I figured I'd mention it.

Any advice, tips or links would be enormously appreciated. I can't for the life of me figure out what's going on.

$\endgroup$
2
  • $\begingroup$ Can I just ask given you are attempting to predict time series data in advance, what is the MSE that you are expecting to be able to reproduce? Have you bench marked this MSE against a simple model such as: 'forecast the 24hr price at the current price'? $\endgroup$
    – Attack68
    Jun 30, 2020 at 15:48
  • $\begingroup$ @Attack68 I don't really have a target MSE, and I haven't benchmarked it against a model like the one you described because I have found that those simple models have a deceptively low MSE (~0.03) (when all variables are normalized to mean=0, sd=1), but are terrible for prediction because they just replicate past values. $\endgroup$ Jun 30, 2020 at 16:06

1 Answer 1

3
$\begingroup$

Some tips:

  1. Do not predict price directly, set your target variable to be (relative) return. as you mentioned your model trains really fast (just 5-9 epochs) and then error does not get lower. I suspect that your neural network just predicts last price + some noise. (you can also see it from your graphs as prediction just lags the true value)

  2. Test your features first on simpler model such as linear regression (or logistic, if you try to predict price direction). Compare results with Neural Network.

  3. run cross validation (split your time series into non overlapping chunks, with some gap in between to insure you do not leak information from the future)

  4. run proper feature selection.(univariate and backward feature selection, for example) Complex models such as neural networks with many hidden layers depend heavily on input feature quality.If you are inputting a lot of non-relevant features, often neural network will not be able to learn.

  5. Set shuffle to false. It is time-series prediction model, shuffling is just leaking future information, making the model over optimistic (i.e. in live set-up prediction of your model will be much worse than back-tested)

$\endgroup$
2
  • $\begingroup$ 1. But how does my neural network predict last price+noise if NONE of my inputs contain the price? They are all exogenous. 3. That's what I have done. The first graph is my out-of-sample prediction, whereas the second graph is my training data fit. 4. Do I do that by removing variables one by one and seeing how model is affected? $\endgroup$ Jun 30, 2020 at 16:10
  • $\begingroup$ 5. Changing the shuffle does not appear to change the results significantly - this is perhaps unsurprising because I don't actually have any time series or lagged inputs. $\endgroup$ Jun 30, 2020 at 16:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.