2
$\begingroup$

I would like to model an index e.g. FTSE 100. I have a list of all companies that make up the index and their stock values (daily high, low, volume, close values). In total, this time series data has 400+ features.

I would like to build a neural network similar to this one, but first I ran a Pearson correlation and found that there is high correlation between stocks.

I want to predict the value of the FTSE 100.

First of all, I scaled the data and then applied PCA to remove correlations and found that 0 components account for 99% of variability. Now my columns look like the following:

Date         FTSEOpen FTSEClose Stock1Open Stock1Close Stock2Open Stock2Close
01/01/2006   2880     2890      144        130         300        333
...
08/01/2018   3862     3851      204        311         134        154

I did PCA on all columns, and got the following result (which doesn't make sense!)

enter image description here

I am currently thinking of adding extra features such as 5day gradients etc. to mix things up.

Also, I'm not sure what Y_TEST is. I understand this is next days data, but I'm still trying to understand what the network input is. If I use all past data, the input dimensions keep increasing with every day.

Let's say I computed PCA, now I have just 1 vector...(1 date column and 1 PCA vector), this now looks like very little data to actually predict stock prices.

Improve this question
$\endgroup$
5
  • $\begingroup$ Why do you have the dependent variables as prices instead of returns? $\endgroup$
    – zsljulius
    Commented Jan 10, 2018 at 2:37
  • $\begingroup$ 0 PCs account for 99% of variance? How can 0 do that? ;) $\endgroup$
    – Richi Wa
    Commented Jan 10, 2018 at 7:49
  • $\begingroup$ @zsljulius Should i write some sort of script to have percentage change, 5 day averages, 5 day low gradient, 5 day integrals etc... to add more useful features? From what you are saying, using prices itself is useless (is it also useless even if I feed it through a Deep Learning Neural Network?) $\endgroup$
    – GRS
    Commented Jan 10, 2018 at 10:58
  • $\begingroup$ @zsljulius If I add extra features, I can easily go above 1000 features, if I do it for all stocks. Then I could technically apply PCA and use these PCA values to train the model instead $\endgroup$
    – GRS
    Commented Jan 10, 2018 at 11:00
  • $\begingroup$ @GRS To do PCA, my understanding is that you want to reduce the dimension of your input features. Say you have 1000 stocks returns, and want to identify several features that are linear combinations of the stock returns to explain the covariance structure. This is a statistical driven way to do factor modeling, unlike that done by Fama French. The goal however is similar. Using return automatically give you the normalization you want, because now they are all in similar scales. $\endgroup$
    – zsljulius
    Commented Jan 11, 2018 at 1:26

1 Answer 1

Reset to default
2
$\begingroup$

Without knowing everything about your procedure, your graph is most probably % cumulative variance explained against each principal component, sorted starting from the largest. See this SE post for a similar plot.

It is conventional to refer to your target vector as $\bf{y}$ so one can only guess that Y_TEST refers to either the labels on the test set (out of sample) or your predictions on the test set.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.