# Tag Info

106

This post is Quant Stack Exchange's master list of data sources. Please append your links to other data sources to the list below. Economic Data See What are the most useful sources of economics data? on Cross Validated. World OECD.StatExtracts includes data and metadata for OECD countries and selected non-member economies. United Kingdom ...

38

Visualization should lead to truth and understanding. As such, I find that simple visualizations tend to be the best. My favorite visualization for showing relationships is the scatterplot. Once you start to even introduce a line plot, you are implying continuities between data that may not exist. And trying to introduce more advanced visualizations like ...

28

NASDAQ makes this information available via FTP and they update it every night. Log into ftp.nasdaqtrader.com anonymously. Look in the directory SymbolDirectory. You'll notice two files: nasdaqlisted.txt and otherlisted.txt. These two files will give you the entire list of tradeable symbols, where they are listed, their name/description, and an indicator as ...

22

I am a big believer in do-it-yourself (DIY) backtesting and data analysis, that is, obtaining your own data and writing your own code. I use my own simple Python scripts to process, test, analyze, and backtest, starting with text-input data files (either OHLC bars or tick data). The reason for DIY: in order to have an effective backtest, analysis, etc., ...

22

Using IBrokers from R is going to be the easiest route. A quick example of capturing data to disk would be: library(IBrokers) tws <- twsConnect() aapl.csv <- file("AAPL.csv", open="w") # run an infinite-loop ( <C-c> to break ) reqMktData(tws, twsSTK("AAPL"), eventWrapper=eWrapper.MktData.CSV(1), file=aapl.csv) ...

22

Consider the standard error, and in particular the distance between the upper and lower limits: $$\Delta = (\bar{x} + SE \cdot \alpha) - (\bar{x} - SE \cdot \alpha) = 2 \cdot SE \cdot \alpha$$ Using the formula for standard error, we can solve for sample size: n = \left(\frac{2 \cdot s \cdot ...

18

Specialised NoSQL database systems are used a lot for time series storage, particularly for tick data: Kx / Kdb is one prominent solution; from Arthur Whitney et al who did A+ at Morgan Stanley Onetick is another, newer entrant, which has traces back to Goldman Sachs Voltdb is something by serial database inventer Michael Stonebreaker SciDB is another ...

17

There is a very good reason why the ratio $$\frac {mean(HIGH-LOW)}{mean(|CLOSE-OPEN|)} \approx 2$$ on various financial series. If the price of a security evolves according to a Wiener process beginning at the opening bell and throughout the day, and the drift is negligible for that period of time, i.e.$\mu=0$, then the denominator of the above ratio ...

16

What do you want to do with the tick data later? Run analytics? You can save tick data to a flat file for all the software cares, but that would be really slow to access later. Instead, you should ideally save the data: Column-oriented - all elements in a field are stored contiguously for better caching Binary - all elements are ready for immediate use; ...

16

The Google Motion Chart is a particularly elegant visualization for 'replaying' time series data. There is also an R package to interface with it.

14

DTN's IQFeed is really good, if a little expensive. I believe it starts at 80 dollars/month and then you add your exchange fees on top. To get access to the developer API you need to pay 300 dollars for a year's worth of access. Details: Real-Time, TRUE Tick-by-Tick Data on US and Canadian Equities (NYSE, NASDAQ, AMEX, Canadian Stock Exchanges) Delayed ...

14

Look at Genesis Trading. Most of the sales guys there are kinda like used car salesmen but they will work with you. Starting up with 50K should not be a problem for them. The offer full depth of book feeds if you are colocated with them. They do offer DMA and you can specify all routing instructions for your orders rather than getting stuck on IBs router. ...

14

Representing time series (esp. tick data) using elaborate data structures may be not the best idea. You may want to try to use two arrays of the same length to store your time series. The first array stores values (e.g. price) and the seconds stores time. Note that the second series is monotonically increasing (or at least non-decreasing), i.e. it's sorted. ...

13

Yahoo rounds the adjusted price to 2 decimals even though dividend amounts often have 3 decimal places. Since they apply the adjustment formula to adjusted prices, if you go far enough back in time, the value they give for Adjusted Price will be different than it would be if there were no rounding. edit: For example, for C (Citigroup), on January 2, 1990, ...

13

There are many specialised products for HF tick data. In addition to KDB which you mentioned, there is OneTick, Vertica, Infobright, and some open-source ones like MonetDB etc. (see http://en.wikipedia.org/wiki/Column-oriented_DBMS). My experience is that Column Oriented Databases are overrated when it comes to tick data, because very often you request the ...

12

Some of us see this as a data-driven, empirical problem. And for Programming with Data, you could do a lot worse than picking R which was made for the task. The CRAN Task View on Finance lists a number of relevant packages. For trading strategies in particular, the quantstrat and blotter packages --which are both still on R-Forge in the TradeAnalytics ...

12

You could try some of tier-2 exchanges which often give data away for free to drum up interest and hence trading volume in their product. You will not get SP500, 10-year note, Crude Oil, Gold, ... futures for free on a tick-by-tick basis simply ... because with per-trade fees down, sales of data are the main revenue for exchanges as sternly noted in the ...

12

If you're planning on analyzing the data later in R, you should take a look at the indexing and mmap packages. Though, as @chrisaycock said, you'll need to save the data in a column-oriented, binary format. If you're downloading the IB data with R, using IBrokers, you can write your own eWrapper to store the data in whatever format you want.

12

Nanex has an interesting way of showing the order-book: The following images show CME's emMni future (S&P 500) depth of book and trades. The images are rainbow (ROYGBIV) color coded by the relative size at each depth level. Red indicates a lot of size, violet indicates size approaching 0. Note that a full minute before each event, the depth starts ...

12

You can either reuse the last computed EMA, or fill-forward the previous period's sample data and recompute the EMA. I generally prefer the second option, which should cause a decay. Only go for the first option if your application won't change its logic based on missing data.

11

I'm only aware about 3 free data sources: EuroNext. Bonds and Equities are available. "Search by Criteria" -> select instrument -> "Data downloads". RBS Databank. Interest rates, FX rate, commodities and CPI GAIN Capital. It contains infomation about FX rates only

11

There was a proxy called the ECU. You should be able to use the weights on the Wikipedia page to get a time series back to 1979. Alternatively, the St. Louis FRED also provides this time series.

11

Bloomberg Open Symbology has this list. Look in the Common Stock precanned file. This will have a bit more data than you probably need as it has a separate entry and unique id for each place an equity is traded. However it is probably the highest quality list available for free anywhere. As for filtering ETFs are broken out in a separate file (Equity_ETP) ...

10

Do not passively use Yahoo where you need reliable historical data; it will just fail at one point (from what I have seen due to corporate actions/dividends not properly implemented). Paying for a single alternative data source will not save you either (Bloomberg sometimes reports crazy intraday prices); the only way is to write some data cleaning routines ...

10

Don't use them. I have used them for years because I couldn't find another source that would provide all stocks in all US exchanges -- till now. But first, about eoddata: their data is very often missing elements, e.g., on any given day the SP500 index data my not be in their data set, even for a normal trading day their ftp files are often out of date, ...

9

Fitch should be available right here: Sovereign Ratings History With Moody's it's not so easy, I don't know if there's a complete source available free of charge. But Sovereign Default and Recovery Rates, 1983-2007 has some data in the appendix III, though not so up to date and in a not that convenient format. Same goes about S&P, Sovereign Ratings ...

9

The first column is just a unique id tagged by Gain; this allows you to separate multiple messages that come in with the same timestamp. D means "dealable": this means that a trade could take place. According to this thread, Gain is known for not dealing around events like major news announcements. [Note: In EBS data, which is much more reliable, "D" ...

8

I don't know how interested you are in the CME data, but I have been learning about options and volatility modeling. I have been working with delayed CME data. I have been able to extract the JSON queries and now have been able to run them in my .NET application to get data for every asset type. Exmaple of ES options data: Run the query below in Chrome ...

8

-- (historical) stock prices -- What do you mean by that? Nominal, real, corrected due to monetary-base-change, corrections with Y-other-things? What is your goal? I have been able to download (historical) stock prices via yahoo and google. Alas looking historical data from Google/Yahoo's screeners can be highly misleading and making conclusion ...

8

I develop strategies for a lot of these different platforms and the one that I feel offers the most is NinjaTrader. It uses C# which is a bit slower than MetaTrader, which if I remember correctly uses a variant of C++, in fact in MT5 there should be almost no difference. However, it makes up for the slowness in spades with the freedom it allows you. Not only ...

Only top voted, non community-wiki answers of a minimum length are eligible