63

I am going to recommend something that I have no doubt will get people completely up in arms and probably get people to attack me. It happened in the past and I lost many points on StackOverflow as people downvoted my answer. I certainly hope people are more open minded in the quant forum. Note - It seems that this suggestion has created some strong ...


25

You are not clear if you want the S&P500 index (SPY), OPRA Options or the Futures. Having spent a lot of time exploring vendors, here is a summary to help you (in alphabetical order): AlgoSeek.com : Intraday data back to 2007 for US Equities, Futures and Options. So you can get S&P 500 data. Intraday they have tick, 1 sec, 1 min and 5 min OHLC bars....


18

That was the devaluation of the French franc on 8 August, 1969 (by 12.5% in terms of par value), decided by President Pompidou and his finance minister Giscard d'Estaing. In those days, exchange rates were quasi-fixed, but subject to periodic realignments (the so-called Bretton Woods system of exchange rates, which was replaced by today's floating rate ...


17

All of the answers above (unfortunately highly upvoted at this point) are missing the point. You shouldn't pick a DBMS or storage solution by general performance benchmarks, you should pick it by use case. If someone says they get a "x ms read", "y inserts per second", "k times speedup", "store n TB data" or "have m years of experience" and use that to ...


13

A friend of mine has created a free source for fundamental financial company data of US listed stocks. All of the data is easily available from https://simfin.com/. Feel free to check it out and send him some feedback, so he can improve the service. Update 30.07.17: So he has just updated the site and now standardized company statements for over 2000 US ...


12

The volatility in the indices long ago was similar in magnitude to what it is today. The problem you are seeing in your plots is one of compounding and scaling. Think of it this way- back in the mid 70's the magnitude of NASDAQ pricing was around \$100. Today it is on the order of \$4000, a change of 40x. In linear terms, a 1% change in the index today (\...


12

The standard answer is going to be that for time series, you want a column store database. These are optimized for range queries (ie: give me everything between two timestamps) because crucially, they store data along one of the dimensions (which you must choose, usually time) contiguously on disk, and thus reads are extremely fast. The alternative, when ...


12

What has changed The API is gone. The new downloads need 1st to parse the Yahoo Finance page to find a hidden crumb and use it as the key for data retrieval over a 2nd URL. The adjusted close is sometimes the adjusted, sometimes is the non-adjusted and sometimes is a different value Lines with literally "null" as the value for the prices have been ...


11

No - clearly you've not seen the licensing agreements the exchanges force you to sign (one way or the other). Generally such firms and individuals have greater utility from the money they'll make working with the data than risking going to jail. Market data is a 5bn / yr business. You're pushing the proverbial up-hill. Anyway, you can get financial index ...


10

Non-disclosure agreements work on the legal side but not in reality, no agreement prevents someone with intent to still steal code or ideas. Protect core code in obfuscated code bases, through APIs installed on the local machine or have it on a server that others do not have access to to and provide access through function calls. Make sure the local machine ...


9

The fx market, contrary to most other asset classes is an almost entirely fragmented over-the-counter market, aside the very small number of fx futures that are trading at dismal liquidity levels. Therefore, you will not encounter a single serious liquidity provider that will take a stab at estimating total traded volume in any of the currency pairs. Having ...


9

There is a deeper issue. Frequentist distributions are not probability distributions because they are designed to be minimax distributions rather than actual distributions. This ignores all of the other problems and this also ignores risk-neutral versus any other measure of risk aversion. An even deeper issue is that these models presume that the ...


8

None of the previous answers have mentioned the fact that Bloomberg supports an API with support for all the main languages (C, C++, Java, Python, Perl -- and even Node and Haskell support on GitHub), on all the relevant operating systems: Windows, Linux, OS X, Solaris. This includes support for tick data which is stored in a rolling window (ie from ...


8

You can get minutely as-traded prices for all US securities on Quantopian, for free. You can't download the original data, but you can query it, analyze it, and do your research within a hosted IPython notebook on the website. Once you register, go to https://www.quantopian.com/research and check the get_pricing() demo notebook. Disclosure: I work at ...


8

I don't see anyplace obvious. My quick reading of Garber's original paper is that he greatly drew on work by Ernst Krelage which I've linked below. The Appendix I of Thompson (2007) describes in detail how he constructs his time series. I think you'll have to get your hands dirty, get into the weeds of these papers. This is probably a good thing, as the ...


7

Let me guess, you fell for one of the fake Quantquote reviews and decided to purchase their buggy data? The reason for the missing quotes is Quantquote data is more of a snap-shot of market activity. It will not record every quote the way TickData or CQG does. ActiveTick is not as expansive as TickData but is more comprehensive than Quantquote. Maybe this ...


7

Markit Pricing Data is a prime source for cds data (not free).


7

To elaborate and emphasize a bit on what @Antoine says, using adjusted prices will be reasonable from a returns point of view, with dividends reinvested. That point, dividend reinvestment, is important because dividend reinvestment itself is a backtesting assumption, namely that dividends could be and would have been invested at the price you have in your ...


7

Some approaches Use only common points - Exclude all holidays in any index. Reduced sample size Loss of information No 'made up' data (consistency) Fill forward - use previous day as you suggested. Issue here is that jumps in the market over holidays are recorded as zero change then a big change. Linear interpolation - linearly interpolate the price ...


7

This issue is incredibly important and I agree there is little practical information about it. To me, the key idea is to find the right matrix completion algorithm that best suits your needs. I work mostly with equity time series and there are substantial missing values issues due to, e.g., as you cite, IPOs with limited history. Recently I have had good ...


6

Depends on your budget of course, but: Mergent offers a great service, expensive though. Six Financial Information offers good Corporate Actions service. (Personally, I would go with this one) Morning Star Interactive Data (very hard company to deal with, their legal would waste a lot of your time, but may still worth to get a quote) Also, consider how ...


6

Both free and paid access to data sets conatianing company financial statement items is available from Quandl. The free data sets are sourced from the SEC based on compnay electronic filings and go back about five years. For example, you could obtain five years of MSFT's quarterly net income using the R call Quandl("RAYMOND/MSFT_NET_INCOME_Q") Lists of ...


6

You can find dataset you need on QUANDL (although some data service is for paying, it is a good data source and not expensive); there are a lot of fundamentals data, surely all you need and you download them in CSV format too. I also know that they provide different frequencies and so even the quarterly frequency. The only problem is that you have to ...


6

Thomson Reuters Tick History. Reasons you might want to use it: 1) It has just about everything you could possibly imagine, going back a long way, and more. 2) The API is (mostly) straightforward to use, so you can download tons of stuff without mucking around in Excel or spreadsheets of any kind. 3) The data can arrive pre-cleaned. Of course, it is ...


6

Morningstar Morningstar partnered with Quantopian, and the latter published the structure of Morningstar's equity fundamentals database: https://www.quantopian.com/help/fundamentals Quantopian users can use this data for free.


6

This is available directly from MSCI's website: MSCI End of Day Index Data Search. If you click on an index name, you can download the entire history.


6

Interesting debate and Not to wake sleeping dogs, the world has moved quite a bit in the 1.5 years, and the data space has exploded. I would like to recommend some new technologies and at the same time share a few of my experiences in this space. As @madilyn is trying to explain: It all depends on your use case. In my experience it's easy to know what you ...


6

To preface, just a minor quibble: French still tracks the momentum anomaly elsewhere on his library, under "Sorts involving Prior Returns"; it's just no longer part of the core FF framework for invalidating the mean-variance-covariance optimized portfolio implied by the CAPM. Originally, Fama-French (FF) developed a three-factor model to invalidate the ...


6

Dukascopy offers historical tick data. Through their historical data website you can download what you want, but registration is required, and lots of manual clicking. However if you are comfortable with scripting, you can directly download the tick data yourself. The URL pattern is http://www.dukascopy.com/datafeed/{currency}/{year}/{month}/{day}/{hour}...


6

In their book "Counterparty Credit Risk, Collateral and Funding" D. Brigo, M. Morini and A. Pallavicini start with a dialogue between a Physics PhD graduate and an experienced practitioner of Quantitative Finance. The topic of P vs Q is presented in that dialogue in a manner meant to be understandable to a new comer. I would certainly recommend you to have ...


Only top voted, non community-wiki answers of a minimum length are eligible