Hot answers tagged

7

You could try Arctic. Other open source column-oriented databases that you may not have considered include LucidDB and C-Store.


4

OpenTSDB is good for large-scale time series storage. metrilyx/opentsdb-pandas and wiktorski/opentsdb_pandas seems to provide the interface with pandas. OpenTSDB and HBase rough performance test | MoreDevs provides a benchmark, may not exactly match your requirements but you can try.


3

Disclosure: I work for the company developing ATSD. Axibase Time Series Database is not open-source but its community edition is free. Time precision is milliseconds. Value is float, double or long. EDIT 1: February 2016. ATSD JDBC Type 4 driver released under Apache 2 license to simplify data access for Java applications. EDIT 2: March 2016. Decimal ...


3

Let me start with a general point: Why do you want to use these datapoints if it is so hard to understand how they are constructed? First of all 4) I am not familiar with testing momentum strategies but you should be aware that the datapoints given are not normal assets you can invest in at the end of the day because at the end of each time period they are ...


3

Cassandra is the obvious choice. With MongoDB or any RDBMS, you will hold all ticks in a table (collection in Mongo-speak) and index by ticker. This means that when you want to retrieve data for a ticker, the data will not be contiguously stored, and you will have a massive usage of index and random reads. Even with SSDs this is slow. For 500k ticks into ...


2

Stock / ETF at 5-minute intervals can be downloaded from Yahoo Finance. See the code below: from urllib import urlretrieve import numpy as np, pandas as pd, sys import datetime as dt, requests import datetime, re, StringIO symbol = sys.argv[1] url='http://chartapi.finance.yahoo.com/instrument/1.0/%s/chartdata;type=quote;range=3d/csv' % symbol response = ...


2

For what concerns Forex data which is, however financial data after all, I often use http://www.histdata.com/. Their data is delivered in .CSV format. For timeframes, I quote the website: We can only deliver you time ordered Tick and M1 (1 minute) data. The data that we have available is organized by forex-pair/year/month. They also provide data for ...


2

There is a times series DBMS (InfiniFlux) that can be easily used with Python. The database is not open source but it does provide a free version for evaluation, too. So you can try whether the DBMS is suitable for your project. You are asking 2M rows should be processed in less than 30 seconds, InfiniFlux can store and retrieve more than 500,000 data ...


2

You're going to want different databases for different data. For instance, the company master, historical data, and fundamental data can probably all live in a standard SQL database (MySQL or Postgres are both reasonable choices). If the intraday data is relatively low-frequency (e.g., 1-minute bars or lower), that can probably be put into the SQL DB as ...


2

An SQL database is generally best for structured data, ad-hoc queries and for queries involving joining several entities together to find the results. It will also help you maintain data consistency and integrity by forcing this more structured design. Recent in-memory features of modern database engines offer most of the remaining performance advantages of ...


1

As @Nicholas said in a comment KX/KDB+ is popular in finance for this purpose. Direct message passing and local aggregation on the machine may be the best method in this case IMO.



Only top voted, non community-wiki answers of a minimum length are eligible