Shannon's entropy for financial times-series (return)

Question

I'm looking at Shannon entropy, and generaly at ways to tell noise from signal when observing intraday returns (at the minute level for now). In python, e.g. I've implemented the fomula (sum of P(xi)*logP(xi) using a numpy histogram.

def rolling_entropy(window):
    cx = np.histogram(window, bins)[0]
    c_normalized = cx/float(np.sum(cx))
    c_normalized = c_normalized[np.nonzero(c_normalized)]
    h = -sum(c_normalized * np.log(c_normalized))  
    return h

The question remains: how to determine the best bin size to digitize the signal? I've assumed the bin size should probably be the tick, and that the number of bins should therefore be variable (max(return)-min(return))/tick size. However, if the number of bins is variable, can the return value be comparable from one time to another (since max and min are likely variable)?

Anyone has a view on this?

Answer 1

What you need is more mutual information rather than Shannon entropy. It is dedicated to capture the influence of one variable on another (you can think about it as a non linear version of Pearson correlations). They are closely related since the mutual information $I$ between two variables $X$ and $Y$ reads: $$I(X;Y) = H(X,Y) - H(X|Y) - H(Y|X)$$ where $H$ is entropy. You have to normalize it (divide by its maximum), see wikipedia for more.

But your question is about binning. You have two solutions:

• keep it discrete (you really want to bin and you think it is a good idea). Hence you need bins to be statistically reliables: use a $\chi^2$ test (it is dedicated to capture the quality of discretization, see for instance this discussion on stats.stackexchange).
• you would like a continuous counterpart of all that: have a look at Kullback–Leibler divergence, it is made for that.

Just have a look at the KL formula between two distributions: $$D_{\mathrm{KL}}(P\|Q) = \int_{-\infty}^\infty p(x) \, \log\frac{p(x)}{q(x)} \, {\rm d}x $$ and compare to Shannon's entropy: $$H_S(X) = -\sum_{i} {\mathrm{P}(x_i) \log_b \mathrm{P}(x_i)}.$$

A last remarks:

• with entropy of discrete variables you have to face numerically bad situation with atoms of small mass (i.e. close to zero, since $\ln(0)$ is not that easy to obtain from a computer). For KL, you will have similar issues to solve when the (empirical) supports of the two distributions are not the same (have a look at the equation).
• there are papers about using such methods on financial time series, like Causality detection based on information-theoretic approaches in time series analysis.