0
$\begingroup$

I believe I have found a bug in Optimal Number of Clusters (ONC) from the paper "Detection of False Investment Strategies Using Unsupervised Learning Methods".

def clusterKMeansTop(corr0,maxNumClusters=10,n_init=10): 
  corr1,clstrs,silh=clusterKMeansBase(corr0,maxNumClusters=corr0.shape[1]-1,n_init=n_init) 
  clusterTstats={i:np.mean(silh[clstrs[i]])/np.std(silh[clstrs[i]]) for i in clstrs.keys()} 
  tStatMean=np.mean(clusterTstats.values()) 
  redoClusters=[i for i in clusterTstats.keys() if clusterTstats[i]<tStatMean] 
  if len(redoClusters)<=2: 
    return corr1,clstrs,silh 
  else: 
    keysRedo=[];map(keysRedo.extend,[clstrs[i] for i in redoClusters]) 
    corrTmp=corr0.loc[keysRedo,keysRedo] 
    meanRedoTstat=np.mean([clusterTstats[i] for i in redoClusters]) 
    corr2,clstrs2,silh2=clusterKMeansTop(corrTmp, \ 
      maxNumClusters=corrTmp.shape[1]-1,n_init=n_init) 
    # Make new outputs, if necessary 
    corrNew,clstrsNew,silhNew=makeNewOutputs(corr0, \ 
      {i:clstrs[i] for i in clstrs.keys() if i not in redoClusters},clstrs2) 
    newTstatMean=np.mean([np.mean(silhNew[clstrsNew[i]])/np.std(silhNew[clstrsNew[i]]) \ for i in 
    clstrsNew.keys()]) 
    if newTstatMean<=meanRedoTstat: 
      return corr1,clstrs,silh 
    else: 
      return corrNew,clstrsNew,silhNew

The line

if newTstatMean<=meanRedoTstat: 

should be changed to:

if newTstatMean<=tStatMean: 

and delete the line:

meanRedoTstat=np.mean([clusterTstats[i] for i in redoClusters]) 

otherwise the algorithm is comparing apples and oranges to maximize expected quality. Because meanRedoTstat is the expected quality of the below-average clusters while newTstatMean is the expected quality of above-average + the below-average re-clustered (with kmeansBase()) clusters. Hence newTstatMean is much more likely to be larger - even if reclustering under-performs current below-average clusters.

In the picture below is an example of the recursion on a 183 x 183 matrix where tstatMean=0.62 and is equal to the below clusters which are indicated with arrow. While tstatMean should be 1.07 Bug in correlation matrix of 183 x 183

$\endgroup$
6
  • 3
    $\begingroup$ I don't clearly see how we could be of help here. Maybe you could contact the authors? $\endgroup$ – Kermittfrog Jan 13 at 13:10
  • $\begingroup$ Thanks for the feedback. I would like some feedback - if you also see that that there is an error in the algorithm - or if im wrong. I have sent the author an email. $\endgroup$ – Endre Moen Jan 13 at 13:13
  • 1
    $\begingroup$ Hi @Endre: Sorry, I cannot give you any feedback on the algo. $\endgroup$ – Kermittfrog Jan 13 at 13:53
  • 2
    $\begingroup$ Not entirely surprising: de Prado's oeuvre is largely a hall of smoke and mirrors. $\endgroup$ – steveo'america Jan 13 at 16:38
  • $\begingroup$ BTW, there are like 5 different methods for fighting overfit given in this paper, any of which seem like more promising candidates for the problem than some unsupervised clustering mumbo-jumbo. $\endgroup$ – steveo'america Jan 13 at 22:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.