evaluate the predictive power of a signal to predict stock price - interview question

Question

I am a young Statistics graduate. A few days ago as an interview question, I have been asked to evaluate the predictive power of a Signal time series (supposedly output by an Artificial Intelligence system) in predicting a stock price time series and I was provided with a sample data set as follows:

Date	Open	High	Low	Close	Adj Close	Signal
202106-01	627.80	633.80	620.55	620.55	623.90	85.11
2021-06-02	620.13	623.36	599.14	620.13	605.12	76.59
2021-06-03	601.80	604.55	571.22	571.22	572.84	68.73
2021-06-04	579.71	600.61	577.20	600.61	599.05	78.47
2021-06-07	591.83	610.00	582.88	610.00	605.13	78.63

and so on for some few weeks. No further information have been provided.

I then pointed out that without knowing the analytical formula of the function employed to predict the stock price from the Signal value

Pred(Signal) = Stock Price Prediction

it was not possible (at least for what I know) to calculate any prediction error and thus to evaluate the prediction power of Signal.

The reply from the interviewer was that the problem is solvable as it is.

FIRST Question: how could one evaluate the predictive power of Signal given only the data above?

As a further question from the interviewer I was asked to "review the quality of the data, list any potential errors, and propose corrected values. Please list each quality check error and correction applied."

Really I had no idea how to proceed in answering this Second Question. SECOND Question Anyone has any pointers on any possible data quality procedure that I could have applied?

Thank you for your kind suggestions.

Answer 1

You might find more traction at https://stats.stackexchange.com/, since most ML and data kind of problems are handled there.

I am no ML expert but let me brainstorm for you, since this kind of task is slightly more an art than a science. You need be creative than just following a det of rules.

FIRST Question: how could one evaluate the predictive power of Signal given only the data above?

This looks like a regression type task, predict k-steps ahead given current or past signals. Question that could have been asked if possible is whether the signal for a particular day is given before or together with the OHLC data. If so, then we want to se if the signal can predict future OHLC data (and not same row). The rest is the usual ML data procedures

Example:

• Explore data for validity etc.
• Divide the data into train/test.
• Normalize columns, and e.g. convert to k-step close price returns.
• Explore e.g. cross correlation between signal and k-step close return. Maybe signal predict not close price return but high-low range? Maybe normalize close return by high-low range? Play around, visualize scatter plots. Easier to predict direction of the future price via e.g. logistic regression?
• Make an e.g. linear regression for Close price return k steps ahead using the current signal. Try for various steps of k. How well does Signal predict Close Price k steps ahead of time: what does the error look like, are there outliers, do we need to change the fitting approach, robust? Does the model predict better than simply guessing zero return? If there is trend in the data, maybe subtract it first.
• Do we need to use previous signals into consideration as well?
• Does it make more sense to predict Close or Adj Close? Possibly the signal isn't taking adjustment (corporate actions) into account.
• Evaluate on test set / perform cross-validation etc.

SECOND Question Anyone has any pointers on any possible data quality procedure that I could have applied?

Some basic things I can think of

• missing values?
• assert 0<low<open<high and 0<low<close<high, dates are increasing without gaps, etc..
• Histogram of columns, histogram of changes in values (returns) etc. Any extreme values (e.g. stock split)? What is the distribution of the signal?
• Normalization applied based on observation.