5
$\begingroup$

I'm looking at Shannon entropy, and generaly at ways to tell noise from signal when observing intraday returns (at the minute level for now). In python, e.g. I've implemented the fomula (sum of P(xi)*logP(xi) using a numpy histogram.

def rolling_entropy(window):
    cx = np.histogram(window, bins)[0]
    c_normalized = cx/float(np.sum(cx))
    c_normalized = c_normalized[np.nonzero(c_normalized)]
    h = -sum(c_normalized * np.log(c_normalized))  
    return h

The question remains: how to determine the best bin size to digitize the signal? I've assumed the bin size should probably be the tick, and that the number of bins should therefore be variable (max(return)-min(return))/tick size. However, if the number of bins is variable, can the return value be comparable from one time to another (since max and min are likely variable)?

Anyone has a view on this?

$\endgroup$
  • $\begingroup$ Just out or curiosity, how will Shannon entropy separate the signal from the noise in financial returns? I've not heard that thesis before. $\endgroup$ – pteetor Nov 11 '15 at 1:41
  • $\begingroup$ @pteetor : I don't know that it does. I've exhumed distant memories about signal processing, read a bit (Kolmogorov entropy is a more powerful concept, but can't be directly calculated). Anyhow, Shannon's entropy is expressing the information content in a signal, so the idea is that a lower value would indicate a direction, trend or something, while a random noise would actually need as many datapoints as the orgininal signal to be fully described. $\endgroup$ – Max F Nov 11 '15 at 5:58
  • 1
    $\begingroup$ In fact, if I recall correctly, the Shannon entropy should bound the strongest lossless compression of the signal. I have done some work on this topic (entropy and timeseries) if you would like to discuss. $\endgroup$ – Joseph Zambrano Nov 12 '15 at 12:46
  • $\begingroup$ @JosephZambrano: I would love that, yes. Unfortunately my rep probably doesn't allow me to pm you but please do if you can so I can give my details, and/or shoot links your research please! $\endgroup$ – Max F Nov 13 '15 at 3:51
  • $\begingroup$ You should probably use np.log2() $\endgroup$ – sten Mar 23 '17 at 23:20
4
$\begingroup$

What you need is more mutual information rather than Shannon entropy. It is dedicated to capture the influence of one variable on another (you can think about it as a non linear version of Pearson correlations). They are closely related since the mutual information $I$ between two variables $X$ and $Y$ reads: $$I(X;Y) = H(X,Y) - H(X|Y) - H(Y|X)$$ where $H$ is entropy. You have to normalize it (divide by its maximum), see wikipedia for more.

But your question is about binning. You have two solutions:

  • keep it discrete (you really want to bin and you think it is a good idea). Hence you need bins to be statistically reliables: use a $\chi^2$ test (it is dedicated to capture the quality of discretization, see for instance this discussion on stats.stackexchange).
  • you would like a continuous counterpart of all that: have a look at Kullback–Leibler divergence, it is made for that.

Just have a look at the KL formula between two distributions: $$D_{\mathrm{KL}}(P\|Q) = \int_{-\infty}^\infty p(x) \, \log\frac{p(x)}{q(x)} \, {\rm d}x $$ and compare to Shannon's entropy: $$H_S(X) = -\sum_{i} {\mathrm{P}(x_i) \log_b \mathrm{P}(x_i)}.$$

A last remarks:

  • with entropy of discrete variables you have to face numerically bad situation with atoms of small mass (i.e. close to zero, since $\ln(0)$ is not that easy to obtain from a computer). For KL, you will have similar issues to solve when the (empirical) supports of the two distributions are not the same (have a look at the equation).
  • there are papers about using such methods on financial time series, like Causality detection based on information-theoretic approaches in time series analysis.
$\endgroup$
  • 1
    $\begingroup$ Thanks a lot @lehalle, I actually got more than I had bargained for. $\endgroup$ – Max F Nov 11 '15 at 9:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.