0
$\begingroup$

I am a young Statistics graduate. A few days ago as an interview question, I have been asked to evaluate the predictive power of a Signal time series (supposedly output by an Artificial Intelligence system) in predicting a stock price time series and I was provided with a sample data set as follows:

Date Open High Low Close Adj Close Signal
202106-01 627.80 633.80 620.55 620.55 623.90 85.11
2021-06-02 620.13 623.36 599.14 620.13 605.12 76.59
2021-06-03 601.80 604.55 571.22 571.22 572.84 68.73
2021-06-04 579.71 600.61 577.20 600.61 599.05 78.47
2021-06-07 591.83 610.00 582.88 610.00 605.13 78.63

and so on for some few weeks. No further information have been provided.

I then pointed out that without knowing the analytical formula of the function employed to predict the stock price from the Signal value

Pred(Signal) = Stock Price Prediction

it was not possible (at least for what I know) to calculate any prediction error and thus to evaluate the prediction power of Signal.

The reply from the interviewer was that the problem is solvable as it is.

FIRST Question: how could one evaluate the predictive power of Signal given only the data above?

As a further question from the interviewer I was asked to "review the quality of the data, list any potential errors, and propose corrected values. Please list each quality check error and correction applied."

Really I had no idea how to proceed in answering this Second Question. SECOND Question Anyone has any pointers on any possible data quality procedure that I could have applied?

Thank you for your kind suggestions.

$\endgroup$
6
  • $\begingroup$ Define Low Close and Adj Close please. $\endgroup$
    – TickaJules
    May 31, 2022 at 13:20
  • $\begingroup$ Low and Close are two separates columns. The formatting of the table was not right, I have corrected it. No information was provided by the interviewer on the columns headers. $\endgroup$
    – MaryM
    May 31, 2022 at 13:31
  • $\begingroup$ Is your interview over or do you still have time to solve the problem? $\endgroup$ May 31, 2022 at 14:25
  • $\begingroup$ @PontusHultkrantz the review is over. I am not in hurry to get an answer. I am asking in case a similar question occurs in a future interview $\endgroup$
    – MaryM
    May 31, 2022 at 15:40
  • $\begingroup$ You should have asked "at what time horizon are you trying to predict?" (eg. one day ahead, 2 days ahead, k-days ahead, to the next open, etc.). If they don't answer you might say "OK, I will assume 1 day"). $\endgroup$
    – nbbo2
    Jun 1, 2022 at 18:43

1 Answer 1

2
$\begingroup$

You might find more traction at https://stats.stackexchange.com/, since most ML and data kind of problems are handled there.

I am no ML expert but let me brainstorm for you, since this kind of task is slightly more an art than a science. You need be creative than just following a det of rules.

FIRST Question: how could one evaluate the predictive power of Signal given only the data above?

This looks like a regression type task, predict k-steps ahead given current or past signals. Question that could have been asked if possible is whether the signal for a particular day is given before or together with the OHLC data. If so, then we want to se if the signal can predict future OHLC data (and not same row). The rest is the usual ML data procedures

Example:

  • Explore data for validity etc.
  • Divide the data into train/test.
  • Normalize columns, and e.g. convert to k-step close price returns.
  • Explore e.g. cross correlation between signal and k-step close return. Maybe signal predict not close price return but high-low range? Maybe normalize close return by high-low range? Play around, visualize scatter plots. Easier to predict direction of the future price via e.g. logistic regression?
  • Make an e.g. linear regression for Close price return k steps ahead using the current signal. Try for various steps of k. How well does Signal predict Close Price k steps ahead of time: what does the error look like, are there outliers, do we need to change the fitting approach, robust? Does the model predict better than simply guessing zero return? If there is trend in the data, maybe subtract it first.
  • Do we need to use previous signals into consideration as well?
  • Does it make more sense to predict Close or Adj Close? Possibly the signal isn't taking adjustment (corporate actions) into account.
  • Evaluate on test set / perform cross-validation etc.

SECOND Question Anyone has any pointers on any possible data quality procedure that I could have applied?

Some basic things I can think of

  • missing values?
  • assert 0<low<open<high and 0<low<close<high, dates are increasing without gaps, etc..
  • Histogram of columns, histogram of changes in values (returns) etc. Any extreme values (e.g. stock split)? What is the distribution of the signal?
  • Normalization applied based on observation.
$\endgroup$
4
  • $\begingroup$ thank you Pontus Hultkrantz for our suggestions! If I understand correctly you are suggesting to perform set of analysis under several assumptions and to ask for additional information. However no additional information will be provided. And the question to be answered is "analyze the signal’s effectiveness or lack thereof in forecasting stock price, using whatever metrics you think are most relevant.". So it is not like the interviewer is asking and open question so that one can choose to perform other kind of analysis just to show of one's skills. The question is very precise. $\endgroup$
    – MaryM
    May 31, 2022 at 15:47
  • $\begingroup$ also thanks a lot for suggesting to post the answer to stats.stackexchange.com $\endgroup$
    – MaryM
    May 31, 2022 at 15:52
  • 1
    $\begingroup$ Then just assume that for eg 2021-06-02, use that signal to predict next day open or close price. Maybe you can use signal and next day open price to predict the close? Is this better than just guessing at the current price? Perform a hypothesis test to evaluate if it is statistically significant. $\endgroup$ May 31, 2022 at 16:26
  • 1
    $\begingroup$ @MaryM IMO the question is quite open ended. There is no universally agreed upon way to evaluate the signal your given. $\endgroup$
    – Bob Jansen
    Jun 2, 2022 at 3:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.