I'm using Python/Pandas. Using naive nested for-loops to do Beta calculation for all ~5k stocks by ~5k days (moving window ~250 days) is unbearably slow. Is there any fast and elegant way to accomplish this goal?

Thanks in advance!

Edit: Simply using Numpy instead of Pandas for all the intermediate steps, would speed up the whole process by >10X.

  • $\begingroup$ You mention 'naive' nested for-loops, what do you suspect would be better? What have you tried? $\endgroup$
    – Phil H
    May 23 '18 at 8:15
  • $\begingroup$ What method are you using to calculate Beta? $\endgroup$
    – amdopt
    May 23 '18 at 14:14
  • $\begingroup$ Consider - in addition to other proposed solution - trying pandas.iteritems() and pandas.iterrows(). These by-column and by-row iterators are pretty awesome. $\endgroup$ May 27 '18 at 20:24

I don’t know how naïve your nested loops are, but I assume you are using the OLS calculation $\beta = (X’X)^{-1}X’Y$, where $X$ contains the index returns and $Y$ contains the security returns.

If you have data for all time periods for all securities, then $(X’X)^{-1}$ will not change for each security. The best solution would be to use numpy to calculate the matrix multiplication directly for all securities. Alternatively, you can calculate $(X’X)^{-1}$ before entering the loop, and then calculate $\beta$ for every individual security.

If you don’t have data for all time periods, then there are speedups, but it gets more complicated.

  • $\begingroup$ Thanks, this method does reduce the amount of calculations. Can it handle empty values well? $\endgroup$
    – sym44
    May 29 '18 at 2:57
  • $\begingroup$ This doesn't really handle empty values, that's why I mentioned missing values in my response. If you have empty values, a good first step for speedups would be to separate your securities into those with complete data and those with incomplete data. If you're lucky, the number of securities with complete data will completely outnumber those with incomplete data. $\endgroup$ May 30 '18 at 16:34

You might find this code snippet helpful. It's the vectorized beta calculation used by Zipline, an open source backtester written in python.

It is computed over a lookback window, with data for all assets over that time period. As Tim mentioned above, this can be efficiently computed using numpy and matrix multiplication.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.