9
$\begingroup$

Apologies if this question is vague, I've gone over how to word it several times in my head, and I'm not sure it gets clearer each time.

I've been looking at this website article https://www.quantstart.com/articles/Basics-of-Statistical-Mean-Reversion-Testing and have been investigating the code for the Hurst exponent in Python. The article gives a code snippet of Python as follows (to calculate Hurst):

def hurst(ts):
    """Returns the Hurst Exponent of the time series vector ts"""
    # Create the range of lag values
    lags = range(2, 100)

    # Calculate the array of the variances of the lagged differences
    tau = [sqrt(std(subtract(ts[lag:], ts[:-lag]))) for lag in lags]

    # Use a linear fit to estimate the Hurst Exponent
    poly = polyfit(log(lags), log(tau), 1)

    # Return the Hurst exponent from the polyfit output
    return poly[0]*2.0

Which works great, but because of my annoying personality where I need to understand something before I use it, I've been driving myself nuts for a day and a half trying to understand how this code has been developed/derived (especially the sqrt(std) part). The article itself does have some brief steps, but I'm not able to follow them. It may be that I don't quite understand the <| ..... |> notation means and how it can relate to standard deviation. Attached here:

Hurst equations

Can anyone provide a link to an article, website or paper that shows from what principles this calculation could have been derived? From Racine's paper I'm aware that Hurst's original method was the RS method, but I believe the method used in the code is from the generalized Hurst exponent or Standard method.

My mathematics isn't at pHd level, but I do have an Engineering degree, so it's not totally useless either. What I am having a huge problem understanding is why the code uses the square root of standard deviation, so if anyone could shed some light on that, it would be greatly appreciated.

Thanks for your time, apologies again if this isn't totally clear.

$\endgroup$
3
  • $\begingroup$ Is the step for calculating the tau vector still clear? $\endgroup$
    – vanguard2k
    Aug 8, 2017 at 10:11
  • $\begingroup$ I understand the part: (subtract(ts[lag:], ts[:-lag])) as this is just subtracting the lagged series from the original time series. What I'm not clear on is why we take the standard deviation of it then the square root of that, as I don't see anything in the equations given in the article (wish I could reproduce them here - but can't figure out how to reproduce formulae here) that indicates why I should be using std and then sqrt. $\endgroup$
    – Wei
    Aug 8, 2017 at 10:16
  • $\begingroup$ It may be that the <| |> sign in those equations is confusing me, as they may have some higher mathematics meaning that I am unaware of. $\endgroup$
    – Wei
    Aug 8, 2017 at 10:21

4 Answers 4

2
$\begingroup$

I am unaware of the notation in this case but I am still trying to make sense of it. (maybe someone can jump in with an edit?) Here's what I have got:

There is the notion of quadratic variation out there that could apply here. You can also think about it as a scalar product (which is the same). You can see that the notation here is rather sloppy, because $\text{log}(t)$ denotes the log price process at time $t$. For discrete points in time, you can interpret it as the vector of log prices at the times $t_i$ and treat the bracket as a scalar product (now denoting $\log(t+\tau)$ and $\log(t)$ as a vector): $$ \langle (\text{log}(t+\tau)-\text{log}(t))^2\rangle \approx \langle \text{log}(t+\tau)-\text{log}(t),\text{log}(t+\tau)-\text{log}(t)\rangle $$

Now, I try to switch to a more precise notation.

Let $t_i, i = 1,\ldots,T$ be the discrete timestamps and $P_i$ the price at $t_i$. Then the term above means the following:

$$ \left(\sum_{i=1}^{T-\tau} (\log(P_{i+\tau})-\log(P_i))^2\right)^{1/2}. $$

Now to determine the hurst exponent $H$, we say that this is approximately $\tau^{2H}$:

$$ \left(\sum_{i=1}^{T-\tau} (\log(P_{i+\tau})-\log(P_i))^2\right)^{1/2} \approx \tau^{2H}$$

taking the logarithm on both sides gives: $$ \log(\ldots) \approx 2H \log(\tau)$$

So what the algorithmdoes is, take 99 values for $\tau$ and calculate the quadratic variation of the lagged differences. Then regress those onto $\tau$ to get an approximation for $2H$.

Now, if we come back to the code: I dont know what the polyfit function does but I assume it performs a polynomial regression with the degree as a parameter. What I dont get is the last line where the result is multiplied by two but you should be able to verify this if you clarify the function output (what is regressed on what, what coefficients are saved where).

$\endgroup$
8
  • 1
    $\begingroup$ Thanks you very much for all the effort that went into this reply. I've spent much time going over your answer and I'm afraid I have 2 follow up questions. I am extremely grateful for the time that you have put into this answer already. 1. If the first equation is the scalar product, would that not give your second equation, without the ^(1/2) or square root? My understanding is that a scalar product of 2 vectors (or the same vector in this case) is simply each term multiplied by itself added together (without the sqrt). Where does the 1/2 come from? $\endgroup$
    – Wei
    Aug 8, 2017 at 13:04
  • $\begingroup$ 2. Apart from the issue mentioned earlier, I understand the progression of all your equations. What I'm unsure of is how this relates to the std term in the Python code, as I can't see anything in your equations that relates to a standard deviation. If it isn't too much to ask, could you quickly clarify the relation? $\endgroup$
    – Wei
    Aug 8, 2017 at 13:06
  • 2
    $\begingroup$ @Wei Of course, I was just trying to offer my thoughts here. :) Of course you are right with respect to the square root. Thats a mistake on my side. Well, the thing is that after taking the log, this will be just a multiplier. Now, if we come to the last line of the code it multiplies the result by two. I am not sure whether the terms in the papers are 100% correct. The standard deviation is the square root of the sum of squares. Then, the code takes the square root again so you have a power of 1/4. After taking the logarithm, this will be a constant again... I know were not there yet but... $\endgroup$
    – vanguard2k
    Aug 8, 2017 at 14:27
  • $\begingroup$ plus, std will subtract a mean but still I think thats what happens... $\endgroup$
    – vanguard2k
    Aug 8, 2017 at 14:33
  • $\begingroup$ You are indeed correct, using the sqrt later comes out (after taking the log) to a multiplier of 1/2, which explains the magical 2.0 multiplication. Which should give the same answer, but sure did add to my confusion. And in terms of the definition for std or variance, again what you say makes sense. What you have in your second equation is in fact the sum of the squares, which is what the variance is, just without the mean being subtracted. $\endgroup$
    – Wei
    Aug 8, 2017 at 16:20
3
$\begingroup$

@vanguard2k I haven't exactly been able to satisfy myself with the derivation of the original code, but I have been able to do the next best thing. I've looked at the source of the code, which is QuantStart which credits Dr Tom Starke, which uses a slightly different code and also credits Dr Ernie Chan. I've then gone to Dr Chan's blog and used his principles to come up with my own code. It uses variance instead of std, gets rid of the sqrt, and uses a 2.0 divisor instead of a 2.0 multiplier (which is the 1/4 that you mention). And the results it gives are the same as the original code. The difference is I understand (I think) the principles that are behind the code below and the final answer is the same. Thanks alot for all your help troubleshooting with me, in the end you were pretty much right, all the elements that didn't seem to make sense cancelled each other out in the end (I guess).

http://epchan.blogspot.fr/2016/04/mean-reversion-momentum-and-volatility.html

  1. Dr Chan states that if z is the log price, then volatility, sampled at intervals of τ, is volatility(τ)=√(Var(z(t)-z(t-τ))). To me another way of describing volatility is standard deviation, so std(τ)=√(Var(z(t)-z(t-τ)))

  2. std is just the root of variance so var(τ)=(Var(z(t)-z(t-τ)))

  3. Dr Chan then states: In general, we can write Var(τ) ∝ τ^(2H) where H is the Hurst exponent

  4. Hence (Var(z(t)-z(t-τ))) ∝ τ^(2H)

  5. Taking the log of each side we get log (Var(z(t)-z(t-τ))) ∝ 2H log τ

  6. [ log (Var(z(t)-z(t-τ))) / log τ ] / 2 ∝ H (gives the Hurst exponent) where we know the term in square brackets on far left is the slope of a log-log plot of tau and a corresponding set of variances.

lags = range(2,100)

def hurst_ernie_chan(p):

variancetau = []; tau = []

for lag in lags: 

    #  Write the different lags into a vector to compute a set of tau or lags
    tau.append(lag)

    # Compute the log returns on all days, then compute the variance on the difference in log returns
    # call this pp or the price difference
    pp = subtract(p[lag:], p[:-lag])
    variancetau.append(var(pp))

# we now have a set of tau or lags and a corresponding set of variances.
#print tau
#print variancetau

# plot the log of those variance against the log of tau and get the slope
m = polyfit(log10(tau),log10(variancetau),1)

hurst = m[0] / 2

return hurst
$\endgroup$
2
$\begingroup$

I think the given python code snippet is composed according to the following steps:

\begin{equation} var(\tau) = \left< |z(t+\tau)-z(t)|^2 \right> \thicksim \tau^{2H} \end{equation}

\begin{equation} \Rightarrow var(...)\thicksim \tau^{2H} \end{equation}

\begin{equation} \Rightarrow std(...) \thicksim \tau^{H} \end{equation}

\begin{equation} \Rightarrow sqrt(std(...)) \thicksim \tau^{\frac H 2} \end{equation}

\begin{equation} \Rightarrow log(sqrt(std(...)) \thicksim \frac H 2 log(\tau) \end{equation}

thus, after applying ployfit(), we get the result H which is poly[0]*2.0

$\endgroup$
1
$\begingroup$

I fail to understand how this code works too.

In the past, I have used this matlab implementation for the generalized hurst exponent calculation and it was quite reliable.

Recently someone has translated this into python, but I haven't tested this yet.

Hope that helps.

$\endgroup$
3
  • $\begingroup$ Thanks for the reply. Unfortunately, it's not so much the code I have a problem with, but understanding how the author has derived the code from the equations I posted. I think a lot of my confusion comes down to the <| |> symbols, do you know what they represent? Thanks. $\endgroup$
    – Wei
    Aug 8, 2017 at 11:51
  • $\begingroup$ As mentioned in this paper: [link]arxiv.org/pdf/1109.0465.pdf 'it denotes the sample average over the time-window' $\endgroup$
    – sen_saven
    Aug 8, 2017 at 12:04
  • $\begingroup$ Thanks for the link to the python code and the paper, it has provided food for thought for me, and I will need to digest it over the coming days. Cheers again. $\endgroup$
    – Wei
    Aug 8, 2017 at 18:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.