1
$\begingroup$

I'm using Python/Pandas. Using naive nested for-loops to do Beta calculation for all ~5k stocks by ~5k days (moving window ~250 days) is unbearably slow. Is there any fast and elegant way to accomplish this goal?

Thanks in advance!

Edit: Simply using Numpy instead of Pandas for all the intermediate steps, would speed up the whole process by >10X.

Improve this question
$\endgroup$
3
  • $\begingroup$ You mention 'naive' nested for-loops, what do you suspect would be better? What have you tried? $\endgroup$
    – Phil H
    May 23, 2018 at 8:15
  • $\begingroup$ What method are you using to calculate Beta? $\endgroup$
    – amdopt
    May 23, 2018 at 14:14
  • $\begingroup$ Consider - in addition to other proposed solution - trying pandas.iteritems() and pandas.iterrows(). These by-column and by-row iterators are pretty awesome. $\endgroup$
    – Igor Pozdeev
    May 27, 2018 at 20:24

2 Answers 2

Reset to default
3
$\begingroup$

I don’t know how naïve your nested loops are, but I assume you are using the OLS calculation $\beta = (X’X)^{-1}X’Y$, where $X$ contains the index returns and $Y$ contains the security returns.

If you have data for all time periods for all securities, then $(X’X)^{-1}$ will not change for each security. The best solution would be to use numpy to calculate the matrix multiplication directly for all securities. Alternatively, you can calculate $(X’X)^{-1}$ before entering the loop, and then calculate $\beta$ for every individual security.

If you don’t have data for all time periods, then there are speedups, but it gets more complicated.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Thanks, this method does reduce the amount of calculations. Can it handle empty values well? $\endgroup$
    – sym44
    May 29, 2018 at 2:57
  • $\begingroup$ This doesn't really handle empty values, that's why I mentioned missing values in my response. If you have empty values, a good first step for speedups would be to separate your securities into those with complete data and those with incomplete data. If you're lucky, the number of securities with complete data will completely outnumber those with incomplete data. $\endgroup$
    – Tim Wilding
    May 30, 2018 at 16:34
3
$\begingroup$

You might find this code snippet helpful. It's the vectorized beta calculation used by Zipline, an open source backtester written in python.

It is computed over a lookback window, with data for all assets over that time period. As Tim mentioned above, this can be efficiently computed using numpy and matrix multiplication.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.