# Data: to clean or not to clean

From risk management point of view using cleaned data (excluding or modifying extreme/outlier observations) would give less conservative measure as compared to real-world raw data. So they are more reluctant to base their conclusions on facts that are really observed.

However, portfolio managers would want to use cleaned data in order to get a more robust and stable estimation of the distribution generating the large majority of the return data.

As a result when they optimize their portfolio using some sort of risk measure (st.dev, CVaR, etc.) these two parties will get totally different picture.

Should having two different approaches be allowed in the first place? If yes, then how conflict situations (e.g. exceeding risk budgets) can be solved? In what cases do you think data cleaning is permissible? Sort of philosophical questions, but would love to know your thoughts.

Update

Thank you all for the answers.

@will, if data is error-free but has some outliers/extreme values, as per the below it seems to be fine to somehow reduce their impact in portfolio optimization:

Application of robust statistics to asset allocation models

Many strategies for asset allocation involve the computation of the expected value and the covariance matrix of the returns of financial instruments. How much of each instrument to own is determined by an attempt to minimize risk — the variance of linear combinations of investments in these financial assets — subject to various constraints such as a given level of return, concentration limits, etc. The covariance matrix contains many parameters to estimate and two main problems arise. First, the data will very likely have outliers that will seriously affect the covariance matrix. Second, with so many parameters to estimate, a large number of return observations are required and the nature of markets may change substantially over such a long period. In this paper we discuss using robust covariance procedures, FAST-MCD, Iterated Bivariate Winsorization and Fast 2-D Winsorization, to address the first problem and penalization methods for the second. When back-tested on market data, these methods are shown to be effective in improving portfolio performance. Robust asset allocation methods have great potential to improve risk-adjusted portfolio returns and therefore deserve further exploration in investment management research.

Robust Portfolio Construction (can't find the full version)

Outliers in asset returns factors are a frequently occurring phenomenon across all asset classes and can have an adverse influence on the performance of mean–variance optimized (MVO) portfolios. This occurs by virtue of the unbounded influence that outliers can have on the mean returns and covariance matrix estimates (alternatively, correlations and variances estimates) that are inputs are optimizer inputs. A possible solution to the problem of such outlier sensitivity of MVO is to use robust estimates of mean returns and covariance matrices in place of the classical estimates of these quantities thereby providing robust MVO portfolios. We show that the differences occurring between classical and robust estimates for these portfolios are such as to be of considerable concern to a portfolio manager. It turns out that robust distances based on a robust covariance matrix can provide reliable identification of multidimensional outliers in both portfolio returns and the exposures matrix of a fundamental factor model, something that is not possible with one-dimensional Winsorization. Multidimensional visualization combined with clustering methods is also useful for returns outlier identification. The question of using robust and classical MVO vs. optimization-based fat-tailed skewed distribution fits and downside risk measure is briefly discussed. Some other applications of robust methods in portfolio management are described, and we point out some future research that is needed on the topic.

Outliers and Portfolio Optimization

In this paper we study the impact of outliers on global minimum variance portfolios. From the method developed by Gomez and Maravall (1997, 2000), we detect and correct outliers in Cac40 French index and in three French stocks included in it. It appears that all financial data present outliers, some of them may be explained by economic and financial events. We calculate the conditional volatility forecast for 60, 120 and 180 business days, using GARCH (1,1) model with 440 observations. As suggested by Franses and Ghijsels (1999), we show that outliers disturb the volatility estimates. Indeed, it seems better to correct outliers before forecasting volatility than used unadjusted series. To evaluate the forecast error, we compute the MSPE and the MAPE. Finally, we examine the impact of outliers on the global variance minimum portfolio structure. The weights of each stocks are significantly different if the series are beforehand ajusted or not. Moreover, portfolio evaluation is better for adjusted data rather than for unadjusted data. Consequently, it seems important to take into account outliers in portfolio optimisation because they affect portfolio variance, weights of portfolio and portfolio evaluation.

There are some other resources, but I do not have full access. Also, I clearly recall where practitioners mentioned taking care of outliers in portfolio construction. Obviously, they were clear and open about their "manipulations" (e.g. threw away 2-3 observations from 2007-08 period). That is why I was wondering if this approach has some merits.

@noob2 - I was looking at clean.boudt as a potential solution for outlier detection and data cleaning.

Any further thoughts on this?

• What do you mean by "cleaned data"? May 26, 2017 at 6:30
• throwing away or decreasing the magnitude of outlier/extreme observations.
– AK88
May 26, 2017 at 6:33
• When you "clean" your data, it should never be about removing things which actually happened because they break your algos, that means your algos are wrong. It should be about removing mistakes, correcting for corporate actions, and removing made up data.
– will
May 26, 2017 at 7:11
• As a portfolio/fund manager I can tell you directly that your second paragraph is completely false. The type of data you are referring to is not "cleaned", it's manipulated for a goal seeking purpose. @will comment is 100% accurate. May 28, 2017 at 14:54
• @amdopt completely agree on the "manipulation" point - if you're doing this internally to make your algos work, then you're basically saying "i am right, the data is wrong", and making an executive decision that you know better than reality. If you have to get yout work validated, they will spot something this obvious (one you certainly hope), and you'll have no way of justifying it.
– will
May 28, 2017 at 15:03

To expand upon what others have said in the comments especially @will's. Data "cleansing" is not just about finding outliers or extreme values or manipulating data.

The examples below are fairly common scenarios from even the largest data vendors that must also be considered for historical or ongoing analysis. All of these issues below can throw an entire model off.

• Vendor provides incorrect values on a stock price or volume.
• Ex: The adjusted close not taking into account splits or dividends.
• Provides a incorrect contract specifications for a corporate bond or swap.
• EX: The wrong coupon frequency, currency code, exc..
• Historical intra-day quotes with incorrect time stamps.
• An exchange stops accepting orders but still sends bid/ask quotes mid market.
• Thank you for your reply. Let's assume - hypothetically - that the data are perfectly fine and have no errors. The detected outliers are real and observed in the market. My PM happens to be using a portfolio construction/analytics software that has a feature that takes care of outliers (winsorization or whatever). She goes ahead and constructs a portfolio using this method as well as the classical method, which accepts data as they are. Goes on to backtesting and reverts back with similar findings as in the existing research (some mentioned in the OP). How should I set my risk limits?
– AK88
May 31, 2017 at 3:34
• en.wikipedia.org/wiki/Winsorizing
– AK88
May 31, 2017 at 3:40