New answers tagged data
Are your 407 stocks all different? No A and B listings contained that are strongly if not perfectly correlated? The observation that the daily covariance matrix is singular makes me wonder. You can try the package corpcor for another shrinkage estimator.
Cassandra is the obvious choice. With MongoDB or any RDBMS, you will hold all ticks in a table (collection in Mongo-speak) and index by ticker. This means that when you want to retrieve data for a ticker, the data will not be contiguously stored, and you will have a massive usage of index and random reads. Even with SSDs this is slow. For 500k ticks into ...
You might want to take a look at some of the supply chain data, I personally have no idea how useful this is. Another interesting area is the import export data sets. There are many interesting papers on this.
Top 50 recent answers are included