So im currently focussing my research on Momentum-Trading Strategies. I downloaded Constitutents of different All Share indices (including Price, Return index, Market value and Dividend Yield). For what types of errors do i have to look with regard to Datastream ?

Are there some common methods which has to be used before working with Datastream data? (i'm using R for my analysis)

There are for example a lot of days where the Price/Return doesn't change at all for like 5 days. Do i have to exclude these values?

Since the extraction of large Datasets in my university takes a lot of time, are there any other sources (not necessarily free) for high quality data requests? (Like Prices and MV of all stocks traded on a particular emerging market stockexchange for the last 30 years). I think the PC in my University would explode if i would request 5k stocks at once.


1 Answer 1



Thomson Reuters Datastream is one of the most commonly used and widely accepted data-source for non-US data in empirical finance. Working with financial data is based on many filters prior to any calculations. My answer focuses especially on "data cleaning" methods for Datastream, which are published in academic journals and commonly used in research papers - so i do not step into basic filters like e.g. Winsorizing and other methods.

Ince/Porter (2006)

This paper analyzes US-data from Datastream with the CRSP database and suggests several data-cleaning methods. The conclude:

We document important issues of coverage, classification, and data integrity and find that naive use of TDS data can have a large impact on econoic inferences.


  • TDS internally rounds prices to the nearest penny which can cause nontrivial differences in the calculated returns when prices are small. As a solution, drop (i.e. set to NA) observations (at the time of your portfolio formation or variable calculations) where the end-of-previous-month price is less than \$1.00 (or in domestic currency).

  • The papers finds many instances of data errors, where prices are far to low and returns therefore are to high. The screen for this type of error by setting to missing / NA any return $R_t$ above 300% that is reversed within one month. If $R_t$ or $R_{t-1}$ is greater than 300% and $(1+R_t)(1+R_{t-1})-1$ is less than 50%, they set $R_t$ and $R_{t-1}$ to missing.

  • Level 1 screening:

    • Remove all non-local firms where data-type GEOG is another one than that of the country you are interested in.
    • Exclude American depository receipts and other non-common equity by eliminating all observations where data-type TYPE is not equity ("EQ").
  • Level 2 screening: This screening requires more effort than level 1 and is based on analyzing data-type NAME and screen for key word or phrases that indicate the security is non common equity (e.g. having "REIT" in the name or a percentage sign which indicates a participatory note).

  • As described in my answer here, you may use "unpadded" data or follow this paper an delete all zero returns from the end of the sample until the first non-zero return.

Schmidt et al. (2011)

Appendix A (especially Table A.2) lists detailed information on data-screening.

  • Country lists: There are (1) Worldscope lists, starting with "WSCOPE" or "WS" and end with a two-letter country code. (2) Research lists start with "F" and ends with a three-to-five letter country code. (3) dead lists begin with "DEAD" and end with a two-letter country code.

The most important screening methods in my opinion are:

  • Screen for major listings on stocks (data-type MAJORshould equal "Y").

  • Screen DS04: Set all returns to missing for which the price is greater than 1,000,000 of the domestic currency.

  • Screen DS09: Set all returns to missing, for which the monthly return is greater than 990%.

Campbell et al. (2010)

In addition to Ince/Porter (2006) level 2 name screening, they suggest the following suspicious word parts: "CV", "CONV", "CVT", "FD", "OPCVM", "PREF", "PF", "PFD", "PFC", "PFCL", "RIGHTS", "RTS", "UNIT", "UNITS", "WTS", "WT", "WARR", "WARRANT", and "WARRANTS".


Serious research is hard work - especially data cleaning and sample constructions. There are many other (often free) sources like Yahoo Finance. However, there are many fundamental errors in these sources, far worse than the flaws mentioned above in Datastream. I may not recommend these free sources for any serious attempt in getting published in (high quality) financial journals.

  • $\begingroup$ Thank you so much for your effort ! Do you think using an index constitution list (like FTSE all share for UK, yearly updated to include dead stocks) instead of a country list, since equity-type is automatically included and very small stocks are excluded, is appropriate? This would reduce a lot of the size of the data to analyze. (Of course the steps are the same but working with it on a regular PC gets faster) $\endgroup$
    – KDMS
    Jul 26, 2019 at 14:05
  • $\begingroup$ I started with the first cited paper but because of a lag of R-experience im struggling with the Return-Reverse-exclusion. My attempt would be to to create a lagged dummy variable for every Return value below 0.5 (Return of -50%) and multiply it with the Returns above 3. Doesn't seem to be an elegant version but i think it would work. Btw cool picture! $\endgroup$
    – KDMS
    Jul 26, 2019 at 14:13
  • $\begingroup$ A last question, is winsorizing appropriate in analyzing trading strategies like Momentum ? The top and bottom deciles are the most important in this case. $\endgroup$
    – KDMS
    Jul 26, 2019 at 15:43
  • $\begingroup$ I don't know if this harms any of the community roles, but: Do you offer code review? Don't get me wrong, i don't want you to write a single line of code for me, i just need someone who would check for any significant errors in my code. By checking with the papers i focus on i know that my statistical codes are correct (had the same data as the researchers) but i don't know if the codes for constructing my research portfolios are correct. If you'd do, i'd send you my Codes and you could look at them and call a price, if not its fine as well, helped me a lot with your answers. $\endgroup$
    – KDMS
    Jul 28, 2019 at 15:10
  • $\begingroup$ Using index constitution list may be appropriate - it all depends on what type of research you are doing and what level you want to achieve. Working with constitution lists gives you a very detailed look on your data-set; just play with simple metrics the first few days - you may profit in the long run if you carefully know your data. Same with winsorizing: Research is about trying different things to get a fully insight - there is no clear "right" or "wrong" approach here. However, if winsorized results are very different from un-winsorized, you may look on why this (surprisingly) happens. $\endgroup$ Jul 29, 2019 at 13:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.