Hot answers tagged

62

I am going to recommend something that I have no doubt will get people completely up in arms and probably get people to attack me. It happened in the past and I lost many points on StackOverflow as people downvoted my answer. I certainly hope people are more open minded in the quant forum. Note - It seems that this suggestion has created some strong ...


61

NASDAQ makes this information available via FTP and they update it every night. Log into ftp.nasdaqtrader.com anonymously. Look in the directory SymbolDirectory. You'll notice two files: nasdaqlisted.txt and otherlisted.txt. These two files will give you the entire list of tradeable symbols, where they are listed, their name/description, and an indicator as ...


43

Consider the standard error, and in particular the distance between the upper and lower limits: \begin{equation} \Delta = (\bar{x} + SE \cdot \alpha) - (\bar{x} - SE \cdot \alpha) = 2 \cdot SE \cdot \alpha \end{equation} Using the formula for standard error, we can solve for sample size: \begin{equation} n = \left(\frac{2 \cdot s \cdot \alpha}{\Delta}\...


27

quandl is a new data source for all kind of econometric time series.


23

Don't use them. I have used them for years because I couldn't find another source that would provide all stocks in all US exchanges -- till now. But first, about eoddata: their data is very often missing elements, e.g., on any given day the SP500 index data my not be in their data set, even for a normal trading day their ftp files are often out of date, ...


20

There is a very good reason why the ratio $$\frac {mean(HIGH-LOW)}{mean(|CLOSE-OPEN|)} \approx 2$$ on various financial series. If the price of a security evolves according to a Wiener process beginning at the opening bell and throughout the day, and the drift is negligible for that period of time, i.e.$\mu=0$, then the denominator of the above ratio ...


17

All of the answers above (unfortunately highly upvoted at this point) are missing the point. You shouldn't pick a DBMS or storage solution by general performance benchmarks, you should pick it by use case. If someone says they get a "x ms read", "y inserts per second", "k times speedup", "store n TB data" or "have m years of experience" and use that to ...


16

Academic access to Thomson Reuters Tick History: www.sirca.org.au The Thomson Reuters Tick History database provides millisecond-timestamped tick data going back to January 1996, covering 45 million OTC and exchange-traded instruments worldwide. The database currently updates at a rate of 1 million messages per second and is around 3 Petabytes uncompressed....


15

Your question is very vague (e.g. what are you trying to measure, and what "tick data" do you have), but I'll give you some pointers: In general, when people consider how prices evolve, they will tend to think about things like volatility and correlation dynamics. So I would start by defining exactly what you want to measure. The irregularity of time ...


14

The reason that "traditional" NoSql databases will not get much up take in finance is that they are designed to solve a different problem. Most NoSql databases from the web world are designed with two central design parameters. First key lookups should be very fast. Second is that operations should be atomic at the row level and should not span records. This ...


14

Slightly ugly bash one-liner for a sorted JSON array: echo "[\"$(echo -n "$(echo -en "$(curl -s --compressed 'ftp://ftp.nasdaqtrader.com/SymbolDirectory/nasdaqlisted.txt' | tail -n+2 | head -n-1 | perl -pe 's/ //g' | tr '|' ' ' | awk '{printf $1" "} {print $4}')\n$(curl -s --compressed 'ftp://ftp.nasdaqtrader.com/SymbolDirectory/otherlisted.txt' | tail -n+2 ...


14

There are many specialised products for HF tick data. In addition to KDB which you mentioned, there is OneTick, Vertica, Infobright, and some open-source ones like MonetDB etc. (see http://en.wikipedia.org/wiki/Column-oriented_DBMS). My experience is that Column Oriented Databases are overrated when it comes to tick data, because very often you request the ...


14

From my experience, EODData is pretty much that you get what you pay for. Its not a very sophisticated product. They email you the files you subscribe to, and thats that. I have had an issue before of where the emails didn't go through anymore and I never heard anything from them. On Quality, I can't make a claim on its accuracy. It seems good. I find issue ...


14

That information is available in the SEC's EDGAR database, though there can be many flavors of "shares outstanding". It is reported quarterly in a company's 10-Q/K, sometimes on a weighted-average basis. If you don't want to get it manually, a service like Bloomberg will let you access the historical levels quite easily, or you can parse the EDGAR XML feeds. ...


14

Accounting is a vital skill if you end up in a managerial position, and unless your career goal is to always be a cog in someone else's clockwork, then you will eventually find yourself in a managerial/senior partnership position even through quant research. I still play a critical role in my firm's quant strategies team, but here's a few things I've had to ...


13

Don't use them. Their data is very spotty. They offer "minute by minute" data on commodities markets and forex which should be technically 24hrs yet it usually starts from 9 or 11 o clock and the "minutes" end around 3 or 4. I emailed their support alias asking why this was the case. I also copied and pasted the opening and closing times of the exchanges and ...


12

The standard answer is going to be that for time series, you want a column store database. These are optimized for range queries (ie: give me everything between two timestamps) because crucially, they store data along one of the dimensions (which you must choose, usually time) contiguously on disk, and thus reads are extremely fast. The alternative, when ...


12

What has changed The API is gone. The new downloads need 1st to parse the Yahoo Finance page to find a hidden crumb and use it as the key for data retrieval over a 2nd URL. The adjusted close is sometimes the adjusted, sometimes is the non-adjusted and sometimes is a different value Lines with literally "null" as the value for the prices have been ...


11

To get a consolidated feed of most of the data feeds here use Quandl. This is free for limited amount of requests per day.


11

I really wouldn't implement time series on my own unless I had a good reason to. AQR uses pandas, almost everyone in R using zoo or xts. I never like multiple parallel arrays, if it breaks everything is broken, plus it gets uglier as you increment data. If you are doing something in C++, why not have an array of structs for each object where you have ...


11

^GSPC is a price index, not a total return index, so it does not include dividends. SPY is an ETF that holds the underlying stocks. When it receives a dividend it keeps it in a cash account (which of course affects the NAV and market value of SPY shares) until the end of the quarter. At that time (on the 3d friday of Mar Jun Sep or Dec) it will pay out the ...


11

The indices have different quoting conventions. The way that a CDS index is traded is that you pay a fixed amount per year for protection in case of default (100 bps for IG, 500 bps for HY) and therefore the contract does not have a zero present value (as it would have if you paid the par spread, like in a fixed for floating interest rate swap). The amount ...


10

Somewhat more economic data can be found at e.g.: The World Bank The United Nations The OECD More financial: The IMF European Union / EFTA / EMU data: Eurostat European Central Bank (financial) Data from these sources is all freely available. You can also play with data from many of these sources using the Google Public Data Explorer.


10

If you are serious about performance and flexibility, you have to take a look at data.table package in R. Here is the crantastic review. It is lighting fast! I think this is the best package addressing performance and memory issues.


10

Non-disclosure agreements work on the legal side but not in reality, no agreement prevents someone with intent to still steal code or ideas. Protect core code in obfuscated code bases, through APIs installed on the local machine or have it on a server that others do not have access to to and provide access through function calls. Make sure the local machine ...


10

There are so many different data providers, and they all end up using slightly different definitions. For Google, it looks like they use Deutsche Börse (Google) as a data source. I can't tell what Yahoo Deutschland is using. I think your real question, though, is why the different data providers have different answers to the same questions. The answers ...


9

Quandl is a free one, with good economic and market data and an API http://www.quandl.com/


9

Although quite simple connected scatterplots can give interesting new insights on how time series perform together: http://steveharoz.com/research/connected_scatterplot/ As an example: Gold vs. S&P 500 from 1970 till today: The green point marks 1970, the red point is today. Every point is a year, moving vertically upwards means rise in the S&P ...


9

In terms of pricing, Zen-Fire seems to be the best "retail" solution. But as you said, you need to be faster, so you can try some faster and more serious options: QuantHouse - CME's Level 1 market data will cost you around 1500 Euros per month. They have points of presence in most local financial centers in Europe (Stockholm, Frankfurt, etc.) so you can ...


9

Don't use eoddata unless you absolutely have to. I purchased long history of global index and forex prices. There are hundreds and hundreds of incorrectly missing or completely wrong prices. For example (big problem with forex), the price is coming along in the 10s, then spkes up to the 1000's for a few days before dropping back down. Repeat over and ...


Only top voted, non community-wiki answers of a minimum length are eligible