I'm using Python/Pandas. Using naive nested for-loops to do Beta calculation for all ~5k stocks by ~5k days (moving window ~250 days) is unbearably slow. Is there any fast and elegant way to accomplish this goal?
Thanks in advance!
Edit: Simply using Numpy instead of Pandas for all the intermediate steps, would speed up the whole process by >10X.
I don’t know how naïve your nested loops are, but I assume you are using the OLS calculation $\beta = (X’X)^{-1}X’Y$, where $X$ contains the index returns and $Y$ contains the security returns.
If you have data for all time periods for all securities, then $(X’X)^{-1}$ will not change for each security. The best solution would be to use numpy to calculate the matrix multiplication directly for all securities. Alternatively, you can calculate $(X’X)^{-1}$ before entering the loop, and then calculate $\beta$ for every individual security.
If you don’t have data for all time periods, then there are speedups, but it gets more complicated.
You might find this code snippet helpful. It's the vectorized beta calculation used by Zipline, an open source backtester written in python.
It is computed over a lookback window, with data for all assets over that time period. As Tim mentioned above, this can be efficiently computed using numpy and matrix multiplication.
