Tag Info

Hot answers tagged

37

Visualization should lead to truth and understanding. As such, I find that simple visualizations tend to be the best. My favorite visualization for showing relationships is the scatterplot. Once you start to even introduce a line plot, you are implying continuities between data that may not exist. And trying to introduce more advanced visualizations like ...


21

Using IBrokers from R is going to be the easiest route. A quick example of capturing data to disk would be: library(IBrokers) tws <- twsConnect() aapl.csv <- file("AAPL.csv", open="w") # run an infinite-loop ( <C-c> to break ) reqMktData(tws, twsSTK("AAPL"), eventWrapper=eWrapper.MktData.CSV(1), file=aapl.csv) ...


21

NASDAQ makes this information available via FTP and they update it every night. Log into ftp.nasdaqtrader.com anonymously. Look in the directory SymbolDirectory. You'll notice two files: nasdaqlisted.txt and otherlisted.txt. These two files will give you the entire list of tradeable symbols, where they are listed, their name/description, and an indicator as ...


21

Consider the standard error, and in particular the distance between the upper and lower limits: \begin{equation} \Delta = (\bar{x} + SE \cdot \alpha) - (\bar{x} - SE \cdot \alpha) = 2 \cdot SE \cdot \alpha \end{equation} Using the formula for standard error, we can solve for sample size: \begin{equation} n = \left(\frac{2 \cdot s \cdot ...


17

There is a very good reason why the ratio $$\frac {mean(HIGH-LOW)}{mean(|CLOSE-OPEN|)} \approx 2$$ on various financial series. If the price of a security evolves according to a Wiener process beginning at the opening bell and throughout the day, and the drift is negligible for that period of time, i.e.$\mu=0$, then the denominator of the above ratio ...


17

Specialised NoSQL database systems are used a lot for time series storage, particularly for tick data: Kx / Kdb is one prominent solution; from Arthur Whitney et al who did A+ at Morgan Stanley Onetick is another, newer entrant, which has traces back to Goldman Sachs Voltdb is something by serial database inventer Michael Stonebreaker SciDB is another ...


16

What do you want to do with the tick data later? Run analytics? You can save tick data to a flat file for all the software cares, but that would be really slow to access later. Instead, you should ideally save the data: Column-oriented - all elements in a field are stored contiguously for better caching Binary - all elements are ready for immediate use; ...


16

The Google Motion Chart is a particularly elegant visualization for 'replaying' time series data. There is also an R package to interface with it.


14

Look at Genesis Trading. Most of the sales guys there are kinda like used car salesmen but they will work with you. Starting up with 50K should not be a problem for them. The offer full depth of book feeds if you are colocated with them. They do offer DMA and you can specify all routing instructions for your orders rather than getting stuck on IBs router. ...


13

Yahoo rounds the adjusted price to 2 decimals even though dividend amounts often have 3 decimal places. Since they apply the adjustment formula to adjusted prices, if you go far enough back in time, the value they give for Adjusted Price will be different than it would be if there were no rounding. edit: For example, for C (Citigroup), on January 2, 1990, ...


12

Nanex has an interesting way of showing the order-book: The following images show CME's emMni future (S&P 500) depth of book and trades. The images are rainbow (ROYGBIV) color coded by the relative size at each depth level. Red indicates a lot of size, violet indicates size approaching 0. Note that a full minute before each event, the depth starts ...


12

You could try some of tier-2 exchanges which often give data away for free to drum up interest and hence trading volume in their product. You will not get SP500, 10-year note, Crude Oil, Gold, ... futures for free on a tick-by-tick basis simply ... because with per-trade fees down, sales of data are the main revenue for exchanges as sternly noted in the ...


12

Representing time series (esp. tick data) using elaborate data structures may be not the best idea. You may want to try to use two arrays of the same length to store your time series. The first array stores values (e.g. price) and the seconds stores time. Note that the second series is monotonically increasing (or at least non-decreasing), i.e. it's sorted. ...


12

There are many specialised products for HF tick data. In addition to KDB which you mentioned, there is OneTick, Vertica, Infobright, and some open-source ones like MonetDB etc. (see http://en.wikipedia.org/wiki/Column-oriented_DBMS). My experience is that Column Oriented Databases are overrated when it comes to tick data, because very often you request the ...


11

If you're planning on analyzing the data later in R, you should take a look at the indexing and mmap packages. Though, as @chrisaycock said, you'll need to save the data in a column-oriented, binary format. If you're downloading the IB data with R, using IBrokers, you can write your own eWrapper to store the data in whatever format you want.


11

You can either reuse the last computed EMA, or fill-forward the previous period's sample data and recompute the EMA. I generally prefer the second option, which should cause a decay. Only go for the first option if your application won't change its logic based on missing data.


11

There was a proxy called the ECU. You should be able to use the weights on the Wikipedia page to get a time series back to 1979. Alternatively, the St. Louis FRED also provides this time series.


10

Do not passively use Yahoo where you need reliable historical data; it will just fail at one point (from what I have seen due to corporate actions/dividends not properly implemented). Paying for a single alternative data source will not save you either (Bloomberg sometimes reports crazy intraday prices); the only way is to write some data cleaning routines ...


10

Don't use them. I have used them for years because I couldn't find another source that would provide all stocks in all US exchanges -- till now. But first, about eoddata: their data is very often missing elements, e.g., on any given day the SP500 index data my not be in their data set, even for a normal trading day their ftp files are often out of date, ...


8

Shane's advice is good. I think it's worth adding the following two techniques not already mentioned: Self-Organizing Maps (SOMs) Seriation (pdf pertaining to R package seriation, but great intro to the topic). They are not explicit visualize techniques, per se. Instead, they are algos that transform underlying data in ways that aim to lead to ...


8

Fitch should be available right here: Sovereign Ratings History With Moody's it's not so easy, I don't know if there's a complete source available free of charge. But Sovereign Default and Recovery Rates, 1983-2007 has some data in the appendix III, though not so up to date and in a not that convenient format. Same goes about S&P, Sovereign Ratings ...


8

The first column is just a unique id tagged by Gain; this allows you to separate multiple messages that come in with the same timestamp. D means "dealable": this means that a trade could take place. According to this thread, Gain is known for not dealing around events like major news announcements. [Note: In EBS data, which is much more reliable, "D" ...


8

The reason that "traditional" NoSql databases will not get much up take in finance is that they are designed to solve a different problem. Most NoSql databases from the web world are designed with two central design parameters. First key lookups should be very fast. Second is that operations should be atomic at the row level and should not span records. This ...


8

In terms of pricing, Zen-Fire seems to be the best "retail" solution. But as you said, you need to be faster, so you can try some faster and more serious options: QuantHouse - CME's Level 1 market data will cost you around 1500 Euros per month. They have points of presence in most local financial centers in Europe (Stockholm, Frankfurt, etc.) so you can ...


7

Google and Yahoo finance have a survivorship bias -- they only include firms that are still around. I know of no free source that provides the data you seek. I get my data from Compustat and CRSP via the Wharton Resource Data Service, but these (or Bloomberg or Reuters) are likely too expensive for an individual. Have you asked your broker if they will sell ...


7

Yahoo's historical data is sometimes missing dividends. For example: http://finance.yahoo.com/q/hp?s=VWINX&a=00&b=1&c=2010&d=11&e=20&f=2011&g=v (VWINX) is missing two dividends for 2011, though it has the ones for 2010. Also, this paper: http://arxiv.org/PS_cache/arxiv/pdf/1105/1105.2956v1.pdf from 2010 reports Yahoo finance ...


7

OpenTick used to have this... alas they are no more. But here's a link to some decent alternatives. http://blog.fosstrading.com/2009/11/opentick-alternatives.html Some have free data options, but I don't believe that any include tick level data for free. If you are in school and have access to WRDS you can get the TAQ (NASDAQ trade and quotes) which is ...


7

Bloomberg Open Symbology has this list. Look in the Common Stock precanned file. This will have a bit more data than you probably need as it has a separate entry and unique id for each place an equity is traded. However it is probably the highest quality list available for free anywhere. As for filtering ETFs are broken out in a separate file (Equity_ETP) ...


6

You can get 1.5 years of dense milisecond tagged forex tick data from here: http://www.truefx.com/?page=downloads (free account required) There are a few problems with it - some missing days and a couple of days with bad data. EUR/USD has an average of 200,000 ticks/day for January 2011. The Dukascopy ticks mentioned in another answer are also good. ...


6

You have a couple of years of free tick by tick FX data in http://www.dukascopy.com/swiss/english/data_feed/historical/ . The easiest way to get the data is to write scripts to download all the files but if you are only playing around I guess you can get a few of them by hand.



Only top voted, non community-wiki answers of a minimum length are eligible