Take the 2-minute tour ×
Quantitative Finance Stack Exchange is a question and answer site for finance professionals and academics. It's 100% free, no registration required.

I have a bunch of time series; i need to clean them before modelling. So far I just know the “filtering/smoothing” method : -Ex: moving average methodology (filter the data with a moving average (filter), then obtain a noise (serie minus filter) and remove data points which correspond to a high noise (i.e with a specific threshold) :

(simple) Example of the moving average filter method with three outliers :

Data and filter : enter image description here Noise and threshold : enter image description here Cleaned data : enter image description here

Do you recommend a specific filter ? do you know a better automatic method ?

share|improve this question
2  
What is sensible depends on your modelling objective and what model you're using. –  user2763361 Jan 26 at 14:52
1  
@user2763361 I totally concur, see my answer below. –  vonjd Jan 26 at 15:21
    
A filter: EMA. Or, a monotone increasing function like $ln(x)$ or $x^{0.5}$. –  user2763361 Jan 26 at 15:29

2 Answers 2

up vote 7 down vote accepted

Not so fast! I think it is of the utmost importance to first examine whether the data points are real outliers, i.e. noise that is contaminating the data, or perhaps the most important pieces of the time series!

For example when you look at US stock market data of the last 50 years and remove only the ten biggest moves because they are outliers you get a completely different time series!

See page 276 of The Black Swan from Nassim Taleb

So you have to be extremely careful and double check all the data points you remove by whatever available method out there!

In general what you consider an outlier also very much depends on the model you are using. So what seems to be an outlier in one model (e.g. a linear model) is part of the package in a more complex model (e.g. a non-linear model). So it is also a matter of experience how to proceed.

So all in all I think there is no easy answer to your question. A good starting point may be the first chapter of the following new book (2013) which is available online:

Outlier analysis by C. Aggarwal

On a more practical note you can use the forecast-package in R in its new version 5.0 from Rob Hyndman. The new version was just released (27/01/2014) and has upgraded functionality for preprocessing time series and outliers:

http://robjhyndman.com/hyndsight/forecast5/

share|improve this answer

The is many techniques for Outliers Detection. I separate them into Global and Local techniques.

-One of the Global techniques I usually use is the Winsorization which consiste on replacing the extremes values on the density distribution by the value corresponding to a certain quantile. For example, you replace all the values bellow the 5% quantile by this one, and all the values above the 95% quantile by this one. This technique could be helpful if you want to exclude a certain period, let say a crisis period, from you data to have only the regular period.

-For local techniques, I would recommend the Local Outlier Factor, I discovered it recently, and I believe that it's a good technique to deal with some unexpected events on the market for a singular day, or for data issues from the data providers.

For automatic methods, the first one is very easy to implement, you just need to specify the quantile threshold, the second one is bit more complicated but there is some implementation in R that could help!

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.