# How to properly calculate the average across multiple correlations?

I'm trying to obtain an average across 3 correlations. Using Python, I obtain these correlations with:

corr = df.apply(lambda s: df.corrwith(s))


which outputs:

   A         B         C
A  1.000000  0.057896  -0.159932
B  0.057896  1.000000  0.581226
C  -0.159932 0.581226  1.000000


The lower triangle of the array is isolated with:

corr.values[np.tril_indices(len(corr))] = np.nan


Now here is where I'd need your help. I'm aware that an arithmetic mean of corr would be the incorrect approach. From this post, there seems to be some preference for "transform each correlation coefficient using Fisher's Z, calculate the mean of the z values, then back-transform to the correlation coefficient".

I'm doing this as follows:

mean_z = np.nanmean(np.arctanh(corr).values)
mean_corr = np.tanh(mean_z)


Is this approach something you agree with and is it correctly implemented?

The goal is to obtain an average correlation across a portfolio.

• Your calculation seems correct, I don't see any obvious errors. If your sample size is large I would not even care with that transformation to be honest. I understand you want to eliminate the bias in correlation estimator? Please note that sample correlation estimator is biased downward, but fisher transformation biases the estimator upwards. I would suggest using other method to correct the bias, Olkin and Pratt method is superior to the Fisher's. link Bear in mind these methods are only valid for normal pdf!
– emot
Aug 2, 2021 at 13:53
• Thanks so much. Yes, one would need to assume normal distribution, which is complicated in a financial timeseries. I've had "divide by 0" errors on occasion when doing Fisher, so may need to revert to arithmetic average or other methods. The sample sizes for all ts correlations are all equal and n=90. Happy to hear further thoughts, and pls feel free to post your comment as answer. Aug 2, 2021 at 14:34

The problem with sample correlation estimator defined as: $$r_{sample} =\frac{\sum\left(X_i - \bar{X}\right)\left(Y_i - \bar{Y}\right)}{\sqrt{\sum\left(X_i-\bar{X}\right)^2\left(Y_i-\bar{Y}\right)^2}}.$$ is that it is biased. The bias is in fact downward i.e. $$r_{sample}$$ tends to be lower than population $$\rho$$. Therefore when we average biased estimator we are keeping the bias.

Olkin and Pratt (1958) suggested unbiased estimator for correlation coefficient: $$r_{corrected}=r_{sample}(1+\frac{1-r_{sample}^2}{2(n-3)})$$ which very accurate and superior to Fisher's (which biases the estimator upwards), according to link. For sample size $$n=90$$ we see that the correction is really small and you can safely ignore the bias and average the correlations without correction.

Some people claim that you should not calculate mean correlation across different pairs of assets. I tend to disagree with that. Below I present two reasonings.

Average correlation for the portfolio

If you want to calculate average correlation for the portfolio then you should take into account portfolio weights. Tierens and Anadu (2004) link proposes a method to calculate average correlation for portfolio: $$p_{av}=\frac{2\sum_{i=1}^{N}\sum_{j>i}^{N}w_i w_j p_{i,j}}{1-\sum_{i=1}^{N}w_i^2}$$

This average correlation has really nice interpretation, if we have two linear portfolios

• one with identical asset's variance and identical correlation between all pairs $$i, j$$ of assets equal to $$p_{av}$$
• second with identical asset's variance but different correlations between pairs $$i, j$$ of assets equal to $$p_{i,j}$$

then the variance of both portfolios are equal and their VaRs are equal as well. From this it is immediate that when average correlation decreases, the variance of the portfolio variance/risk decreases as well. Therefore average correlation provides useful information.

Measure of similarity of two correlation matrices

We can calculate distance between two correlation matrices and compare how similar they are link. The distance metric is: $$d = 1 - \frac{\text{tr}(R_1 \cdot R_2)}{\|R_1\| \cdot \|R_2\|},$$ where $$R_1$$ and $$R_2$$ are two correlation matrices and the norm is the Frobenius norm. This metric take values from 0 (identical matrix) to 1. We can compare any correlation matrices with that metric. But it turns out if we constrain ourselves into scalars only, then simple mean of all correlations minimizes the distance $$d$$! i.e. $$R_2$$ with off diagonal entries equal to $$p_{av-equal}$$ is most similar to the original matrix $$R_1$$.

$$p_{av-equal}=\frac{\sum_{i=1}^{N}\sum_{j>i}^{N}p_{i,j}}{N(N-1)/2}$$ it's a simple mean of off-diagonal entries.

• Beautifully explained, thanks a million Aug 3, 2021 at 14:37
• The question was asked using python code, it should be answered using python code. Dec 13, 2021 at 2:02