I'm somewhat new into the world of trading algo's, so bare with me. I've made a dataframe with 5 features say. I used preprocessing.scale to scale it. I checked the csv dump of it and it looks fine and dandy. Now, say I've just got todays open, high, low, close, volume data. I've calculated the indicators for it (used a 20 day history to get them). I'd excpect that I would have to scale these new input features so that they'd somewhat match up to what the model was trained with. How can I do that with just 1 row of numbers? Or should I scale the whole 20 day history and hope it works out close to the data scaled for training?
1
-
$\begingroup$ I think in this case the question doesn't have to be closed although it's indeed related to developing a trading strategy. @noob2 if you put your answer in a comment I consider it up-vote worthy. $\endgroup$– Bob Jansen ♦Jan 25, 2017 at 18:24
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
Think of whatever scaling operation that you're using as a model whose parameters were fitted on historical data. The estimated parameters are static.
For example, say one of your features has a minimum value of -3 and maximum value of 5 in your 20 day history, so min-max scaling of 0-1 would take the form:
$\hat{f}: x \mapsto \dfrac{x-x_{min}}{x_{max}-x_{min}},\ x_{min}=-3,\ x_{max}=5$
And you can handle new data points $x$. If you encounter a new feature value of 11 in the future, $\hat{f}\left(11\right)=1.75$. Whether it's valid for your value to fall outside $\left[0,1\right]$ is another issue that depends on your model assumptions.
The other answer before me details the other issue pretty well - how you let the history grow is both a statistical and algorithm design problem. Back to the example I've given, you could let a mix-max scaler scale indefinitely as it requires both $\mathbb{O}\left(1\right)$ memory and amortized $\mathbb{O}\left(1\right)$ time.
$\begingroup$
$\endgroup$
How do you implement a trading algorithm based on a predictive model? The algorithm is driven by a finite set of past data points, that I will call The History.
Each day you observe new data in the financial markets (for example OHLC for that day). You scale this data by reference to the history that you have (for example you compare today's move to the standard deviation of historical points to conclude that "today is a 2 standard deviation up move"). After this you make a prediction again with reference to historical values. Then the recently observed OHLC is added to the History File.
It is up to you whether you let the history file grow indefinitely (eventually it will have some rather stale data) or whether you drop the oldest data point when you add the newest (sometimes called a moving window approach).
When you trained your algo, you should have done the same thing, i.e. scale each point using only past data points (scale $x_t$ based on $x_{t-1},x_{t-2},\cdots$). Ideally the training should be done similar to the live operation, without lookahead.
