How to properly calculate the average across multiple correlations?

Question

I'm trying to obtain an average across 3 correlations. Using Python, I obtain these correlations with:

corr = df.apply(lambda s: df.corrwith(s))

which outputs:

   A         B         C
A  1.000000  0.057896  -0.159932
B  0.057896  1.000000  0.581226
C  -0.159932 0.581226  1.000000

The lower triangle of the array is isolated with:

corr.values[np.tril_indices(len(corr))] = np.nan

Now here is where I'd need your help. I'm aware that an arithmetic mean of corr would be the incorrect approach. From this post, there seems to be some preference for "transform each correlation coefficient using Fisher's Z, calculate the mean of the z values, then back-transform to the correlation coefficient".

I'm doing this as follows:

mean_z = np.nanmean(np.arctanh(corr).values)
mean_corr = np.tanh(mean_z)

Is this approach something you agree with and is it correctly implemented?

The goal is to obtain an average correlation across a portfolio.

Answer 1

The problem with sample correlation estimator defined as: $$r_{sample} =\frac{\sum\left(X_i - \bar{X}\right)\left(Y_i - \bar{Y}\right)}{\sqrt{\sum\left(X_i-\bar{X}\right)^2\left(Y_i-\bar{Y}\right)^2}}.$$ is that it is biased. The bias is in fact downward i.e. $r_{sample}$ tends to be lower than population $\rho$. Therefore when we average biased estimator we are keeping the bias.

Olkin and Pratt (1958) suggested unbiased estimator for correlation coefficient: $$r_{corrected}=r_{sample}(1+\frac{1-r_{sample}^2}{2(n-3)})$$ which very accurate and superior to Fisher's (which biases the estimator upwards), according to link. For sample size $n=90$ we see that the correction is really small and you can safely ignore the bias and average the correlations without correction.

Some people claim that you should not calculate mean correlation across different pairs of assets. I tend to disagree with that. Below I present two reasonings.

Average correlation for the portfolio

If you want to calculate average correlation for the portfolio then you should take into account portfolio weights. Tierens and Anadu (2004) link proposes a method to calculate average correlation for portfolio: $$p_{av}=\frac{2\sum_{i=1}^{N}\sum_{j>i}^{N}w_i w_j p_{i,j}}{1-\sum_{i=1}^{N}w_i^2}$$

This average correlation has really nice interpretation, if we have two linear portfolios

• one with identical asset's variance and identical correlation between all pairs $i, j$ of assets equal to $p_{av}$
• second with identical asset's variance but different correlations between pairs $i, j$ of assets equal to $p_{i,j}$

then the variance of both portfolios are equal and their VaRs are equal as well. From this it is immediate that when average correlation decreases, the variance of the portfolio variance/risk decreases as well. Therefore average correlation provides useful information.

Measure of similarity of two correlation matrices

We can calculate distance between two correlation matrices and compare how similar they are link. The distance metric is: $$d = 1 - \frac{\text{tr}(R_1 \cdot R_2)}{\|R_1\| \cdot \|R_2\|},$$ where $R_1$ and $R_2$ are two correlation matrices and the norm is the Frobenius norm. This metric take values from 0 (identical matrix) to 1. We can compare any correlation matrices with that metric. But it turns out if we constrain ourselves into scalars only, then simple mean of all correlations minimizes the distance $d$! i.e. $R_2$ with off diagonal entries equal to $p_{av-equal}$ is most similar to the original matrix $R_1$.

$$p_{av-equal}=\frac{\sum_{i=1}^{N}\sum_{j>i}^{N}p_{i,j}}{N(N-1)/2}$$ it's a simple mean of off-diagonal entries.