Questions tagged [data]
Questions about handling, obtaining, generating, or analyzing all types of financial or economic data. Please use a more accurate tag if possible; for instance: tick-data, fundamentals, market-data, option-data, ticker-mapping, etc.
500 questions
109
votes
7
answers
204k
views
Where to download list of all common stocks traded on NYSE, NASDAQ and AMEX?
I have a very basic data question: how to get a list of all common stocks traded on NYSE, NASDAQ and AMEX? I would need to be able to get the approximate list of common stocks as is available in ...
- 1,091
94
votes
10
answers
53k
views
Building Financial Data Time Series Database from scratch
My company is starting a new initiative aimed at building a financial database from scratch.
We would be using it in these ways:
Time series analysis of: a company's financial data (ex: IBM's total ...
- 1,043
94
votes
14
answers
22k
views
Innovative ways of visualizing financial data
Finance is drowning in a deluge of data. Humans are not very good at comprehending large amounts of data. One way out may be visualization.
Traditional ways of visualizing patterns, complexities and ...
- 27.7k
50
votes
4
answers
8k
views
How much data is needed to validate a short-horizon trading strategy?
Suppose one has an idea for a short-horizon trading strategy, which we will define as having an average holding period of under 1 week and a required latency between signal calculation and execution ...
- 13.6k
43
votes
8
answers
11k
views
What broker/feed/APIsetup allows for recording the most accurate data (cheaply)?
I'm currently using IB's Java API and getting feeds through them. However the real-time feed is updated only every 250ms and the historical feed only every second.
I'm primarily looking for ES data ...
37
votes
7
answers
25k
views
Usage of NoSQL storage in Finance
I am wondering if anyone has used NoSQL (mongodb, cassandra, etc.) to store and analyze data.
I tried searching the web but was not able to see if the financial firms had gotten in to using nosql ...
- 866
33
votes
9
answers
40k
views
What is the best data structure/implementation for representing a time series?
I was wondering what is best practice for representing elements in a time series, especially with large amounts of data. The focus/context is in a back testing engine and comparing multiple series.
...
- 591
33
votes
5
answers
14k
views
What is a good broker for HFT?
Currently I trade trough IB. I run my HFT strategies (100 roundtrips per hour) but I think that latency is killing me and my profits are shrinking. I need the fastest possible brokers out there which ...
- 2,707
32
votes
9
answers
27k
views
How should I store tick data?
How should I store tick data? For example, if I have an IB trading account, how should I download and store the tick data directly to my computer? Which software should I use?
- 461
30
votes
11
answers
19k
views
Training set of tick-by-tick data?
I'm looking to find a free source of tick by tick data (<1sec) for training purposes. It doesn't need to be longer than a day, and I don't care what instrument, or exchange, or time it is. I just ...
- 401
29
votes
9
answers
20k
views
Any known bugs with Yahoo Finance adjusted close data ?
Yahoo Finance allows you to download tables of their daily historical stock price data.
The data includes an adjusted closing price that I thought I might use to calculate daily log returns as a ...
- 523
27
votes
8
answers
5k
views
What kind of basic framework or application do you use to run your trading algorithms?
I heard about MetaTrader from http://www.metaquotes.net. Is there any other framework or program available? Do you use different software for backtracking and running your trading algorithms?
Thank ...
24
votes
20
answers
27k
views
Is "eoddata" a good data source?
Not sure if this is a relevant question for site, but I am looking to move to www.eoddata.com as my data source.
If anyone has used it, can you tell me how the data quality is ?
I am currently ...
- 1,563
22
votes
4
answers
9k
views
data on historical stock price of bankrupt companies
does anybody know a site where I can download historical data on stocks including companies that have gone bankrupt such as lehman brothers?
it appears that bankrupt companies no longer appear in the ...
- 605
21
votes
11
answers
16k
views
Free intra-day equity data source
Are there any free data source for historical US equity data? Yahoo finance has daily prices but I'm looking for something more granular and goes back 2 or more years (doesn't have to be close to tick ...
- 357
19
votes
3
answers
9k
views
R: How feasible is it to store -- and work with -- tick data in a database connected to R?
I'm looking to convert some tickdata .csv files into a database on a local disk and then use R to call the data and do my various analytics and modelling.
What are some best practices / ...
- 437
19
votes
2
answers
6k
views
How to update an exponential moving average with missing values?
Say you have an Exponential Moving Average being continuously updated over a time series using 1-second-long time periods. What should happen if there is no value for the next second, e.g. there were ...
- 1,805
19
votes
4
answers
23k
views
How to get list of all CUSIPS/ISIN?
I want a list of all CUSIPs/ISINs. It would be nice if they were also categorized (e.g. Bonds/Funds etc). Where can I get such a data?
- 241
19
votes
2
answers
4k
views
Do you have historical tick data you want to donate?
Do you have historical market/pricing ticket data that you would like to donate to the Open Source Trader project (OST)??
Please: upload your files! Once we gather some data, we'll do our best to ...
- 751
17
votes
2
answers
1k
views
Why is the ratio of Hi-Low range to Open-Close range close to 2?
I tried it in several symbols and timeframes with the same result:
$$\frac {mean(HIGH-LOW)}{mean(|CLOSE-OPEN|)}$$
...
- 273
17
votes
4
answers
5k
views
How to generate synthetic FX data for backtesting?
I want to generate synthetic forex data for the purpose of backtesting my trading algorithms. I have some rough ideas in mind on how to do this:
Start with a curve representing a trend, then randomly ...
- 271
16
votes
9
answers
40k
views
Data source for historical Share Outstanding totals for individual stocks?
Data is normally adjusted for splits/reverse splits, etc. The current shares outstanding is usually available. Is there a data repository that captures the shares outstanding for any point in the ...
16
votes
4
answers
14k
views
Analyzing tick data
What are some of the commonly used techniques to analyze tick data?
I am looking at tick data to see how the quotes/ mid-price evolves due to certain events in the market. Since tick data is ...
- 1,563
16
votes
3
answers
6k
views
Free market data (delayed or snapshot)
I know there are similar questions but I don't think there are any identical ones.
Basically, I'm looking for one of these two things.
Delayed market data feed. A tick by tick feed, but delayed by ...
- 261
15
votes
5
answers
11k
views
Best way to store hourly/daily options data for research purposes
There are quite a few discussions here about storage, but I can't find quite what I'm looking for.
I'm in need to design a database to store (mostly) option data (strikes, premiums bid / ask, etc.). ...
- 989
15
votes
3
answers
22k
views
Why the difference between SPY and ^GSPC?
Look at SPY vs ^GSPC -- the difference seems bigger than can be explained by the ETF fees. Is it only because of SPY re-invests dividends quarterly or something else? Since ...
- 305
15
votes
3
answers
2k
views
Effective Euro-USD (EURUSD) Exchange Rate Prior to Euro's Existence
Motivation: I am running a quantitative analysis that requires long-term, exchange rate data.
Problem: Does anyone have methods for dealing with the EURUSD exchange rate prior to the Euro's existence?...
- 153
14
votes
3
answers
2k
views
Literature on generating synthetic time series for testing
I have some market data (daily time series) for bond prices and CDS indices and I would like to generate synthetic versions of these which are statistically "similar" for testing trading strategies. ...
- 241
14
votes
1
answer
5k
views
Volume or Dollar bars vs. volatility normalized and demeaned financial time series
In his book - Advances in Financial Machine Learning, Marcos Lopez de Prado familiarises the reader with a number of ways of normalizing our financial time series data. Below I provide a couple of ...
- 305
14
votes
6
answers
10k
views
Earnings and valuation data sources online
Are there any free/cheap sources for historical data on company earnings and valuations? I can get historical price data from Google and Yahoo, and it looks like I can get about five years of ...
- 1,381
13
votes
6
answers
19k
views
Are public historical time series available for ratings of sovereign debt?
The nice list of free online data sources What data sources are available online? does not mention any data from ratings agencies.
Are historical time series available for sovereign credit ratings (...
- 243
13
votes
2
answers
7k
views
Is Yahoo! Finance data good or bad now?
I have read various blog posts and reddit threads and so on talking about the recent changes to the Yahoo! Finance API and it leaves me very confused. I'm confused about what has changed. Supposedly ...
- 233
13
votes
1
answer
2k
views
Where can I find data on the interbank lending market?
Where can I find disaggregated interbank lending data (i.e. bank A lends to bank B x money at y rate)? I could only find data on interest rates.
I would accept LIBOR market data as well as any ...
- 572
13
votes
1
answer
728
views
What time are Bloomberg Open Symbology Files updated daily?
Does anyone know what time the Bloomberg Open Symbology precanned files are updated every day? I am planning on upating my view of them at 9 AM EST every morning. Furthermore, will the files for ...
- 1,805
13
votes
2
answers
6k
views
Why use a column database for tick/bar data?
I often hear that column-oriented databases are the best choice method for storing time series data in finance applications. Especially by people selling expensive column-oriented databases.
Yet, at ...
- 1,427
12
votes
4
answers
467
views
Seeking Historical Non-Finance Datapoints for Backtesting
I'm working on some financial analysis code which I'd like to test against a historical dataseries to analyze the correlations to my algorithm to some non-finance related data. Ideally, I'd like to ...
- 221
12
votes
7
answers
4k
views
Which data sources are available for cryptocurrencies?
I am primarily looking for price and market cap histories for cryptocurrencies like Bitcoin, Ethereum, zcash, ...
What are some good data sources and APIs where I can download history as well as get ...
- 486
12
votes
1
answer
3k
views
What's the most efficient way to store options and time series data for backtesting?
I would like to know what database would you guys use for storing around 500GB-1TB of options and time series tick data.
The idea is to use it for backtesting so it would have to be as efficient as ...
- 223
12
votes
7
answers
2k
views
When hiring a quant, how can I protect my IP?
I am a one-man operation, and would like to hire a quant for around 4 weeks or work. I am worried that the person I hire might copy my data or the indicators that I have him work on.
What have ...
- 123
12
votes
2
answers
4k
views
Why don't data sources use ISIN instead of symbol?
I've been messing around with stock data off and on for a year or two and just recently found out about the ISIN. Up until I found some data from Siblis Research, none of the other data sources used ...
- 121
12
votes
3
answers
2k
views
Why isn't all market data free?
I am hoping any answers can be threaded somewhere between "don't be naive" and starting a quasi-political flame.
Transparency is suppose to be good for markets and also a social good. The market ...
12
votes
7
answers
9k
views
How to download all 10-K reports for all companies listed on S&P 500?
I am doing a regression analysis of all companies listed on s&p 500. It requires their 10-k reports. Where can I download all of them once?
- 649
12
votes
6
answers
12k
views
Where to get historical intraday stock data?
I hate to ask questions that have been asked before. But I am afraid that this is one of them. I have searched the web for days now, read so many forum posts. But I can't find an answer. Most answers ...
- 121
12
votes
1
answer
387
views
Is there a standard / methodology to determine and grade the quality of OHLC data?
Inputs are most important to any decision making.
For strategy backtesting, the OHLC data is one of the most inputs.
So to ensure the correctness and integrity of OHLC data, we have checks on the ...
- 121
11
votes
3
answers
9k
views
Red Black Trees for Limit Order Book
Why do people suggest using red black trees/balanced binary trees for the levels in a limit order book?
Why are they algorithmically ideal?
- 2,662
11
votes
4
answers
3k
views
Real-time & Fast S&P 500 E-Mini Futures (ES) Data
I trade on local exchanges in Europe, but my HFT strategies need S&P 500 e-mini futures data (ES). I don't need to trade ES, but I need real-time data and I want to have it as fast as possible.
I'...
- 659
11
votes
2
answers
2k
views
Is there a technique for using xts or zoo objects with options data (i.e., many entries per date) in R?
I am starting to work with options data from optionmetrics. I use data frames, but it seems like xts or zoo objects are the way to go for features and speed. I can't figure out the best work-around to ...
- 4,501
11
votes
2
answers
504
views
How to "uncluster" a set of financial data?
I am attempting to evaluate and compare the profit factor of different "test runs" of a FOREX trading strategy.
My problem is that, despite an average time between orders of 2hr+, some of these runs ...
- 303
11
votes
1
answer
1k
views
A generic limit order book: What are the most important queries it should be able to answer?
Assume a class LimitOrderBook which represents a limit order book in a trading system.
To be able to represent the limit order book a data handler reads a feed ...
- 2,129
10
votes
3
answers
4k
views
Algorithm to detect the aggressor side of a trade
Most of the exchanges provide aggressor side property of trades (e.g. Tag=5797 AggressorSide on CME) in their raw data. But many data providers do not provide this information via their datafeed API's....
- 1,022