How to forecast high-frequency data?

Question

Introduction:
I have seen a plenty of articles/books regarding volatility forecasting applied to high frequency data, but none of them were dedicated to forecasting the actual prices (for example bid/ask of currency pairs, stock prices).
Example:
Forecasting Volatility using High Frequency Data (P. R. HANSEN, A. LUNDE)

I realize that forecasting hft data must be hard because of several problems:

• How long model will be valid?
• How to check such validity?
• Will a new model be calculated on time? (NN and most of subtypes of NN need a lot of time to be recalculated)

Questions:
Given the above mentioned problems/restrictions I would like to ask you:

1. What's the best way of forecasting high-frequency data?
   
2. What kind of scientific papers/books should I follow in order to be up-to-date in this topic?
   
   

Personal feedback:
From my own research based on hft data I can write that generally only few models are better than naive forecasting (but only in some sub time series - few minutes etc.)

• Kalman Filtering (calibrated accordingly)
• ARIMA, ARMA models

Generally worse than naive forecasting are:

• Holts linear model (ARIMA (0,2,2)
• Linear Regression
• Exponential Smoothing
• Moving averages (different types)

Answer 1

A good place to start would be to understand the environment in which HFT takes place. What causes prices to form on the lowest possible level?

What mechanisms do exchanges use to process orders, how would they affect the price discovery process?

This in generally known as market-microstructure, and there is a lot of literature available on the subject, but you should also augment that with just physical observation of the orderbook, trades and the resulting price movements to get a better understanding of the whole process.

Once you have an understanding of what you are dealing with, you can start to build and test hypothesis.

Answer 2

at HF it is probably more important to guess the direction (up/down) than actual prices. For that you will have a lot of noise and financial time series have low signal to noise ratio. that is why classifiers (up/down) are the best practice in HF, not punctual predictors of prices. This has also to do with the computational complexity required to forecast a classifier vs a punctual value (simply put, in HF you need to cut down running time, thus computational complexity, in my experience many strategies fail due to implementation time and computational complexity). In any case, Kalman filters, ARMA, PCA, Fourier analysis are all good methods to extract signal. Once you have extracted the signal (i.e. removed most of the noise) then you should move on with other kind of models.