Hot answers tagged

57

NASDAQ makes this information available via FTP and they update it every night. Log into ftp.nasdaqtrader.com anonymously. Look in the directory SymbolDirectory. You'll notice two files: nasdaqlisted.txt and otherlisted.txt. These two files will give you the entire list of tradeable symbols, where they are listed, their name/description, and an indicator as ...


51

I am going to recommend something that I have no doubt will get people completely up in arms and probably get people to attack me. It happened in the past and I lost many points on StackOverflow as people downvoted my answer. I certainly hope people are more open minded in the quant forum. Note - It seems that this suggestion has created some strong ...


42

Consider the standard error, and in particular the distance between the upper and lower limits: \begin{equation} \Delta = (\bar{x} + SE \cdot \alpha) - (\bar{x} - SE \cdot \alpha) = 2 \cdot SE \cdot \alpha \end{equation} Using the formula for standard error, we can solve for sample size: \begin{equation} n = \left(\frac{2 \cdot s \cdot \alpha}{\Delta}\...


26

quandl is a new data source for all kind of econometric time series.


22

Don't use them. I have used them for years because I couldn't find another source that would provide all stocks in all US exchanges -- till now. But first, about eoddata: their data is very often missing elements, e.g., on any given day the SP500 index data my not be in their data set, even for a normal trading day their ftp files are often out of date, ...


21

Specialised NoSQL database systems are used a lot for time series storage, particularly for tick data: Kx / Kdb is one prominent solution; from Arthur Whitney et al who did A+ at Morgan Stanley Onetick is another, newer entrant, which has traces back to Goldman Sachs Voltdb is something by serial database inventer Michael Stonebreaker SciDB is another ...


20

Nanex has an interesting way of showing the order-book: The following images show CME's emMni future (S&P 500) depth of book and trades. The images are rainbow (ROYGBIV) color coded by the relative size at each depth level. Red indicates a lot of size, violet indicates size approaching 0. Note that a full minute before each event, the depth starts ...


20

There is a very good reason why the ratio $$\frac {mean(HIGH-LOW)}{mean(|CLOSE-OPEN|)} \approx 2$$ on various financial series. If the price of a security evolves according to a Wiener process beginning at the opening bell and throughout the day, and the drift is negligible for that period of time, i.e.$\mu=0$, then the denominator of the above ratio ...


19

Bloomberg Open Symbology has this list. Look in the Common Stock precanned file. This will have a bit more data than you probably need as it has a separate entry and unique id for each place an equity is traded. However it is probably the highest quality list available for free anywhere. As for filtering ETFs are broken out in a separate file (Equity_ETP) ...


18

I don't know how interested you are in the CME data, but I have been learning about options and volatility modeling. I have been working with delayed CME data. I have been able to extract the JSON queries and now have been able to run them in my .NET application to get data for every asset type. Exmaple of ES options data: Run the query below in Chrome ...


16

Academic access to Thomson Reuters Tick History: www.sirca.org.au The Thomson Reuters Tick History database provides millisecond-timestamped tick data going back to January 1996, covering 45 million OTC and exchange-traded instruments worldwide. The database currently updates at a rate of 1 million messages per second and is around 3 Petabytes uncompressed....


15

Do not passively use Yahoo where you need reliable historical data; it will just fail at one point (from what I have seen due to corporate actions/dividends not properly implemented). Paying for a single alternative data source will not save you either (Bloomberg sometimes reports crazy intraday prices); the only way is to write some data cleaning routines (...


14

There are many specialised products for HF tick data. In addition to KDB which you mentioned, there is OneTick, Vertica, Infobright, and some open-source ones like MonetDB etc. (see http://en.wikipedia.org/wiki/Column-oriented_DBMS). My experience is that Column Oriented Databases are overrated when it comes to tick data, because very often you request the ...


14

That information is available in the SEC's EDGAR database, though there can be many flavors of "shares outstanding". It is reported quarterly in a company's 10-Q/K, sometimes on a weighted-average basis. If you don't want to get it manually, a service like Bloomberg will let you access the historical levels quite easily, or you can parse the EDGAR XML feeds. ...


14

Your question is very vague (e.g. what are you trying to measure, and what "tick data" do you have), but I'll give you some pointers: In general, when people consider how prices evolve, they will tend to think about things like volatility and correlation dynamics. So I would start by defining exactly what you want to measure. The irregularity of time ...


14

All of the answers above (unfortunately highly upvoted at this point) are missing the point. You shouldn't pick a DBMS or storage solution by general performance benchmarks, you should pick it by use case. If someone says they get a "x ms read", "y inserts per second", "k times speedup", "store n TB data" or "have m years of experience" and use that to ...


13

The reason that "traditional" NoSql databases will not get much up take in finance is that they are designed to solve a different problem. Most NoSql databases from the web world are designed with two central design parameters. First key lookups should be very fast. Second is that operations should be atomic at the row level and should not span records. This ...


13

Accounting is a vital skill if you end up in a managerial position, and unless your career goal is to always be a cog in someone else's clockwork, then you will eventually find yourself in a managerial/senior partnership position even through quant research. I still play a critical role in my firm's quant strategies team, but here's a few things I've had to ...


12

Slightly ugly bash one-liner for a sorted JSON array: echo "[\"$(echo -n "$(echo -en "$(curl -s --compressed 'ftp://ftp.nasdaqtrader.com/SymbolDirectory/nasdaqlisted.txt' | tail -n+2 | head -n-1 | perl -pe 's/ //g' | tr '|' ' ' | awk '{printf $1" "} {print $4}')\n$(curl -s --compressed 'ftp://ftp.nasdaqtrader.com/SymbolDirectory/otherlisted.txt' | tail -n+2 ...


12

From my experience, EODData is pretty much that you get what you pay for. Its not a very sophisticated product. They email you the files you subscribe to, and thats that. I have had an issue before of where the emails didn't go through anymore and I never heard anything from them. On Quality, I can't make a claim on its accuracy. It seems good. I find issue ...


12

Don't use them. Their data is very spotty. They offer "minute by minute" data on commodities markets and forex which should be technically 24hrs yet it usually starts from 9 or 11 o clock and the "minutes" end around 3 or 4. I emailed their support alias asking why this was the case. I also copied and pasted the opening and closing times of the exchanges and ...


12

What has changed The API is gone. The new downloads need 1st to parse the Yahoo Finance page to find a hidden crumb and use it as the key for data retrieval over a 2nd URL. The adjusted close is sometimes the adjusted, sometimes is the non-adjusted and sometimes is a different value Lines with literally "null" as the value for the prices have been ...


11

To get a consolidated feed of most of the data feeds here use Quandl. This is free for limited amount of requests per day.


11

I really wouldn't implement time series on my own unless I had a good reason to. AQR uses pandas, almost everyone in R using zoo or xts. I never like multiple parallel arrays, if it breaks everything is broken, plus it gets uglier as you increment data. If you are doing something in C++, why not have an array of structs for each object where you have ...


11

^GSPC is a price index, not a total return index, so it does not include dividends. SPY is an ETF that holds the underlying stocks. When it receives a dividend it keeps it in a cash account (which of course affects the NAV and market value of SPY shares) until the end of the quarter. At that time (on the 3d friday of Mar Jun Sep or Dec) it will pay out the ...


11

The standard answer is going to be that for time series, you want a column store database. These are optimized for range queries (ie: give me everything between two timestamps) because crucially, they store data along one of the dimensions (which you must choose, usually time) contiguously on disk, and thus reads are extremely fast. The alternative, when ...


11

The indices have different quoting conventions. The way that a CDS index is traded is that you pay a fixed amount per year for protection in case of default (100 bps for IG, 500 bps for HY) and therefore the contract does not have a zero present value (as it would have if you paid the par spread, like in a fixed for floating interest rate swap). The amount ...


10

Somewhat more economic data can be found at e.g.: The World Bank The United Nations The OECD More financial: The IMF European Union / EFTA / EMU data: Eurostat European Central Bank (financial) Data from these sources is all freely available. You can also play with data from many of these sources using the Google Public Data Explorer.


10

If you are serious about performance and flexibility, you have to take a look at data.table package in R. Here is the crantastic review. It is lighting fast! I think this is the best package addressing performance and memory issues.


10

Non-disclosure agreements work on the legal side but not in reality, no agreement prevents someone with intent to still steal code or ideas. Protect core code in obfuscated code bases, through APIs installed on the local machine or have it on a server that others do not have access to to and provide access through function calls. Make sure the local machine ...


Only top voted, non community-wiki answers of a minimum length are eligible