Hot answers tagged data
30
Visualization should lead to truth and understanding. As such, I find that simple visualizations tend to be the best. My favorite visualization for showing relationships is the scatterplot. Once you start to even introduce a line plot, you are implying continuities between data that may not exist. And trying to introduce more advanced visualizations like ...
17
There is a very good reason why the ratio
$$\frac {mean(HIGH-LOW)}{mean(|CLOSE-OPEN|)} \approx 2$$
on various financial series. If the price of a security evolves according to a Wiener process beginning at the opening bell and throughout the day, and the drift is negligible for that period of time, i.e.$\mu=0$, then the denominator of the above ratio ...
16
The Google Motion Chart is a particularly elegant visualization for 'replaying' time series data. There is also an R package to interface with it.
16
Using IBrokers from R is going to be the easiest route. A quick example of capturing data to disk would be:
library(IBrokers)
tws <- twsConnect()
aapl.csv <- file("AAPL.csv", open="w")
# run an infinite-loop ( <C-c> to break )
reqMktData(tws, twsSTK("AAPL"),
eventWrapper=eWrapper.MktData.CSV(1),
file=aapl.csv)
...
16
Specialised NoSQL database systems are used a lot for time series storage, particularly for tick data:
Kx / Kdb is one prominent solution; from Arthur Whitney et al who did A+ at Morgan Stanley
Onetick is another, newer entrant, which has traces back to Goldman Sachs
Voltdb is something by serial database inventer Michael Stonebreaker
SciDB is another ...
16
Consider the standard error, and in particular the distance between the upper and lower limits:
\begin{equation}
\Delta = (\bar{x} + SE \cdot \alpha) - (\bar{x} - SE \cdot \alpha) = 2 \cdot SE \cdot \alpha
\end{equation}
Using the formula for standard error, we can solve for sample size:
\begin{equation}
n = \left(\frac{2 \cdot s \cdot ...
15
What do you want to do with the tick data later? Run analytics? You can save tick data to a flat file for all the software cares, but that would be really slow to access later.
Instead, you should ideally save the data:
Column-oriented - all elements in a field are stored contiguously for better caching
Binary - all elements are ready for immediate use; ...
14
Look at Genesis Trading. Most of the sales guys there are kinda like used car salesmen but they will work with you. Starting up with 50K should not be a problem for them. The offer full depth of book feeds if you are colocated with them.
They do offer DMA and you can specify all routing instructions for your orders rather than getting stuck on IBs router.
...
11
Representing time series (esp. tick data) using elaborate data structures may be not the best idea.
You may want to try to use two arrays of the same length to store your time series. The first array stores values (e.g. price) and the seconds stores time. Note that the second series is monotonically increasing (or at least non-decreasing), i.e. it's sorted. ...
11
There was a proxy called the ECU.
You should be able to use the weights on the Wikipedia page to get a time series back to 1979. Alternatively, the St. Louis FRED also provides this time series.
11
NASDAQ makes this information available via FTP and they update it every night. Log into ftp.nasdaqtrader.com anonymously. Look in the directory SymbolDirectory. You'll notice two files: nasdaqlisted.txt and otherlisted.txt. These two files will give you the entire list of tradeable symbols, where they are listed, their name/description, and an indicator as ...
11
There are many specialised products for HF tick data. In addition to KDB which you mentioned, there is OneTick, Vertica, Infobright, and some open-source ones like MonetDB etc. (see http://en.wikipedia.org/wiki/Column-oriented_DBMS).
My experience is that Column Oriented Databases are overrated when it comes to tick data, because very often you request the ...
10
You could try some of tier-2 exchanges which often give data away for free to drum up interest and hence trading volume in their product.
You will not get SP500, 10-year note, Crude Oil, Gold, ... futures for free on a tick-by-tick basis simply ... because with per-trade fees down, sales of data are the main revenue for exchanges as sternly noted in the ...
10
You can either
reuse the last computed EMA, or
fill-forward the previous period's sample data and recompute the EMA.
I generally prefer the second option, which should cause a decay. Only go for the first option if your application won't change its logic based on missing data.
10
Yahoo rounds the adjusted price to 2 decimals even though dividend amounts often have 3 decimal places. Since they apply the adjustment formula to adjusted prices, if you go far enough back in time, the value they give for Adjusted Price will be different than it would be if there were no rounding.
edit: For example, for C (Citigroup), on January 2, 1990, ...
8
Nanex has an interesting way of showing the order-book:
The following images show CME's emMni future (S&P 500) depth of book and trades. The images are rainbow (ROYGBIV) color coded by the relative size at each depth level. Red indicates a lot of size, violet indicates size approaching 0. Note that a full minute before each event, the depth starts ...
8
If you're planning on analyzing the data later in R, you should take a look at the indexing and mmap packages. Though, as @chrisaycock said, you'll need to save the data in a column-oriented, binary format.
If you're downloading the IB data with R, using IBrokers, you can write your own eWrapper to store the data in whatever format you want.
8
Shane's advice is good. I think it's worth adding the following two techniques not already mentioned:
Self-Organizing Maps (SOMs)
Seriation (pdf pertaining to R package seriation, but great intro to the topic).
They are not explicit visualize techniques, per se. Instead, they are algos that transform underlying data in ways that aim to lead to ...
8
The first column is just a unique id tagged by Gain; this allows you to separate multiple messages that come in with the same timestamp.
D means "dealable": this means that a trade could take place. According to this thread, Gain is known for not dealing around events like major news announcements.
[Note: In EBS data, which is much more reliable, "D" ...
8
In terms of pricing, Zen-Fire seems to be the best "retail" solution. But as you said, you need to be faster, so you can try some faster and more serious options:
QuantHouse - CME's Level 1 market data will cost you around 1500 Euros per month. They have points of presence in most local financial centers in Europe (Stockholm, Frankfurt, etc.) so you can ...
8
Don't use them. I have used them for years because I couldn't find another source that would provide all stocks in all US exchanges -- till now. But first, about eoddata:
their data is very often missing elements, e.g., on any given day the SP500 index data my not be in their data set, even for a normal trading day
their ftp files are often out of date, ...
7
Google and Yahoo finance have a survivorship bias -- they only include firms that are still around. I know of no free source that provides the data you seek. I get my data from Compustat and CRSP via the Wharton Resource Data Service, but these (or Bloomberg or Reuters) are likely too expensive for an individual.
Have you asked your broker if they will sell ...
7
OpenTick used to have this... alas they are no more.
But here's a link to some decent alternatives.
http://blog.fosstrading.com/2009/11/opentick-alternatives.html
Some have free data options, but I don't believe that any include tick level data for free.
If you are in school and have access to WRDS you can get the TAQ (NASDAQ trade and quotes) which is ...
7
The reason that "traditional" NoSql databases will not get much up take in finance is that they are designed to solve a different problem. Most NoSql databases from the web world are designed with two central design parameters. First key lookups should be very fast. Second is that operations should be atomic at the row level and should not span records. This ...
6
You can get 1.5 years of dense milisecond tagged forex tick data from here: http://www.truefx.com/?page=downloads (free account required)
There are a few problems with it - some missing days and a couple of days with bad data.
EUR/USD has an average of 200,000 ticks/day for January 2011.
The Dukascopy ticks mentioned in another answer are also good. ...
6
You have a couple of years of free tick by tick FX data in http://www.dukascopy.com/swiss/english/data_feed/historical/ . The easiest way to get the data is to write scripts to download all the files but if you are only playing around I guess you can get a few of them by hand.
6
Just for future reference, if you are student or academic, you can request for market data on http://www.quantnet.com/forum/.
Many of our members are Wall Street practitioners and as a policy, they will provide such data to help with your research (hence students/academic only). I have been the conduit for many of such transaction in the past.
You will need ...
6
I agree with Shane; I seriously doubt you're going to find publicly available swaption data for free. You might get some sample data with a textbook, or from a published journal article.
If you only need one example, you can find one in the documentation for the BermudanSwaption function in the RQuantLib R package.
6
I really wouldn't implement time series on my own unless I had a good reason to. AQR uses pandas, almost everyone in R using zoo or xts.
I never like multiple parallel arrays, if it breaks everything is broken, plus it gets uglier as you increment data. If you are doing something in C++, why not have an array of structs for each object where you have ...
6
Here's a link to daily weather data. It looks like it goes as far back as the 1940s. There's a link to a CSV file at the bottom of the page. It will only give you one year's worth of data at a time, so you'll have to manually download several files.
http://www.wunderground.com/history/airport/KNYC/2011/3/13/CustomHistory.html
Only top voted, non community-wiki answers of a minimum length are eligible
