Why does Hierarchical Risk Parity ignore the clusters generated?

Question

I am currently working through the Hierarchical Risk Parity algorithm (Lopez de Prado (2016) link ) and trying to understand each of the steps.

I have completed the step of creating the clusters, and visualised them using a dendrogram. Here's an example dendrogram for sake of illustration:

At this point, the way I thought HRP worked was by calculating the min-variance weights in each cluster from the bottom to the top. In the example dendrogram above, this would thus be:

1. Calculate minvar weights between JPM and BoA
2. Calculate minvar weights between JPM+BoA cluster (var calculated using the weights from #1) and BRK
3. Calculate minvar weights between JPM+BoA+BRK cluster and Exxon
4. Etc.

The final weights would then be the product of all the weights calculated in these steps.

I don't seem to be the only one with this mental heuristic. In the documentation for the PyPortfolioOpt library, the following "rough overview" is presented (my highlights):

From a universe of assets, form a distance matrix based on the correlation of the assets. Using this distance matrix, cluster the assets into a tree via hierarchical clustering. Within each branch of the tree, form the minimum variance portfolio (normally between just two assets). Iterate over each level, optimally combining the mini-portfolios at each node.

Seemingly this is not how the HRP algorithm works. Instead, it seems to purely use the clusters to sort the assets in the quasi-diagonalisation step. It then takes this sorted list and creates its own new clusters by bisection, and then calculates minvar working from top to bottom.

In the example dendrogram, this would mean some assets we clustered in the first step would never end up in a cluster together after being sorted. E.g. Facebook and Alphabet would end up being separated in the first bisection step.

Have I understood this correctly? If so, why does this make sense? What is an intuitive way to understand why this works?

Answer 1

This turns out to be a general drawback of the HRP algorithm, as pointed out by Pfitzinger, J., & Katzke, N. (2019) (my highlights):

As shown in Figure 2.3, the naive bisection rule can violate the intuitive character of the result, by placing similar assets into separate clusters for allocation purposes. While centered bisection yields a symmetric allocation tree, which results in well-diversified portfolio weights, the method does not respect empirical cluster boundaries and discards information about the hierarchical structure inferred in the cluster algorithm. Figure 2.3 (top-left) demonstrates the concern, with the bisection separating two closely related assets.

The authors go on to propose an alternative method that does take into account the information contained in the clusters (by recursing into the actual clusters rather than naively bisecting), however their results are not consistently better than naive HRP across returns, variance, turnover and concentration metrics.