I've been tasked with imputing missing values for a dataset of ca. 4000 firms and 225 key metrics (e.g. revenue, net income, EPS, PE etc.). Since I haven't found a thread on here which answers my question I'm just going to ask myself:

Firstly, the purpose of this task is to eventually impement a ML alogrithm which lets me categorize firms that perform better than the market (buy), similar to the market (hold) and worse than the market (sell).

Since this is my first time doing something like this in this kind of scale I'm unsure which imputation method I should use. I have looked at mean/median/most frequent value but this method seems to be too simplistic. Additionally, I have implemented a KNN imputer (from the scikit-learn package) but I find it very hard to asses whether this gives me good results or not.

Note: the implementation of the ML algo is in a jupyter notebook (python) and the data I'm referring to is available on kaggle: https://www.kaggle.com/cnic92/200-financial-indicators-of-us-stocks-20142018

Thanks for any hints or suggestions

  • 2
    $\begingroup$ I want to echo @Mattthew Gunn's concern about adding unavailable data: `` You're going to have to use judgement. There are situations where treating missing values as 0 would be insane. In general, I'd be immensely cautious. '' You don't know why some numbers are missing and replacing them incorrectly can actually turn unprofitable strategies into profitable ones. Thus, your backtest may be completely misleading (i.e. dangerously erroneous). Work with the data you have. Instead of meddling with the data, try to get better quality data. $\endgroup$ – Kevin Mar 25 at 9:26
  • $\begingroup$ I second @Kevin 's sentiment. DO NOT impute data. Financial data analysis is hard enough without you adding all kinds of bias and error by imputing data. You're better to exclude those years / datapoints entirely. $\endgroup$ – R110 Mar 25 at 10:20
  • $\begingroup$ Firstly, thanks for your comments. While, in general, I agree with you, simply removing the NAs is not possible mainly for two reasons: (i) e.g. for the year 2017 I have a data frame with dimensions 4000 (firms) x 225 (variables) but over 225k NAs hence this would result in a huge data wasting preventing me from doing any analysis whatsoever and (ii) neither do I have the possibility to gather any more information in order to circumvent having to deal with NAs altogether. $\endgroup$ – AtK42 Mar 25 at 11:14
  • $\begingroup$ I would look more carefully at the files and check if there are specific factors (columns) and specific companies that contribute to the N/A count in a disproportionate way. You might reduce your N/A count substantially if you drop those outliers. For example, drop row (ticker) it has more than 10 N/As, or if the row doesn't have a required value, e.g. revenue. $\endgroup$ – Sergei Rodionov Mar 25 at 13:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.