time series data modeling for deep learning

Question

what is the best format to feed the input data, which are time series with varying density over time, to a deep learning network, while at any iteration we want to feed a batch of data including a historical background?

Is it better to consider a constant size of data records or a constant time window including variable data record size? Or is there a better way?

Answer 1

I am not sure I perfectly understand your question, the concept of "time series with varying density over time" is not very clear.

One thing is for sure, the optimal way to "feed" a neural network is a function of the type of NNet itself and of the learning method you have chosen.

For time series

• either you believe your data are iid vectors, and you can use a fully connected perceptron
• either they are auto-correlated, and probably an LSTM (Long-Short Time Memory) is a good option to consider
• if you expect them to have multi-scale patterns ConvNets are probably a good choice.

For the learning, the way you feed the NNet has to be compatible with your previous choice:

• fully connected perceptrons on iid observations should be trained via Stochastic Gradient Descent (SGD), ie using random small batches
• LSTM have to be feed by long blocks of consecutive data,
• ConvNets have to be feed by blocks preserving the structure of your expected patterns.

Here are few remarks about your last point about using a fixed window in number of observations versus an observation duration:

• you need to have a decent number of observations in each batch, hence if you choose a duration, take it long enough
• you need to not break the type of pattern you expect the NNet to capture; take your duration (or number of points) long enough.

In essence, for mini-batches driving a SGD, their composition has to be chosen such that the estimated gradient (stemming from the averaging over the mini-batch), has the same variance from a mini-batch to the other. Look at your data and make the best choice!