# Bug found in Optimal Number of Clusters algorithm - from de Prado and Lewis (2018)

I believe I have found a bug in Optimal Number of Clusters (ONC) from the paper "Detection of False Investment Strategies Using Unsupervised Learning Methods".

def clusterKMeansTop(corr0,maxNumClusters=10,n_init=10):
corr1,clstrs,silh=clusterKMeansBase(corr0,maxNumClusters=corr0.shape[1]-1,n_init=n_init)
clusterTstats={i:np.mean(silh[clstrs[i]])/np.std(silh[clstrs[i]]) for i in clstrs.keys()}
tStatMean=np.mean(clusterTstats.values())
redoClusters=[i for i in clusterTstats.keys() if clusterTstats[i]<tStatMean]
if len(redoClusters)<=2:
return corr1,clstrs,silh
else:
keysRedo=[];map(keysRedo.extend,[clstrs[i] for i in redoClusters])
corrTmp=corr0.loc[keysRedo,keysRedo]
meanRedoTstat=np.mean([clusterTstats[i] for i in redoClusters])
corr2,clstrs2,silh2=clusterKMeansTop(corrTmp, \
maxNumClusters=corrTmp.shape[1]-1,n_init=n_init)
# Make new outputs, if necessary
corrNew,clstrsNew,silhNew=makeNewOutputs(corr0, \
{i:clstrs[i] for i in clstrs.keys() if i not in redoClusters},clstrs2)
newTstatMean=np.mean([np.mean(silhNew[clstrsNew[i]])/np.std(silhNew[clstrsNew[i]]) \ for i in
clstrsNew.keys()])
if newTstatMean<=meanRedoTstat:
return corr1,clstrs,silh
else:
return corrNew,clstrsNew,silhNew


The line

if newTstatMean<=meanRedoTstat:


should be changed to:

if newTstatMean<=tStatMean:


and delete the line:

meanRedoTstat=np.mean([clusterTstats[i] for i in redoClusters])


otherwise the algorithm is comparing apples and oranges to maximize expected quality. Because meanRedoTstat is the expected quality of the below-average clusters while newTstatMean is the expected quality of above-average + the below-average re-clustered (with kmeansBase()) clusters. Hence newTstatMean is much more likely to be larger - even if reclustering under-performs current below-average clusters.

In the picture below is an example of the recursion on a 183 x 183 matrix where tstatMean=0.62 and is equal to the below clusters which are indicated with arrow. While tstatMean should be 1.07

• I don't clearly see how we could be of help here. Maybe you could contact the authors? – Kermittfrog Jan 13 at 13:10
• Thanks for the feedback. I would like some feedback - if you also see that that there is an error in the algorithm - or if im wrong. I have sent the author an email. – Endre Moen Jan 13 at 13:13
• Hi @Endre: Sorry, I cannot give you any feedback on the algo. – Kermittfrog Jan 13 at 13:53
• Not entirely surprising: de Prado's oeuvre is largely a hall of smoke and mirrors. – steveo'america Jan 13 at 16:38
• BTW, there are like 5 different methods for fighting overfit given in this paper, any of which seem like more promising candidates for the problem than some unsupervised clustering mumbo-jumbo. – steveo'america Jan 13 at 22:10