3
$\begingroup$

I don't work in finance, but I've stumbled upon a problem that you guys may have to deal with in your jobs.

My problem is a random walk in high dim spaces ( > 100), in which I'm looking for vectors that explore a limited portion of that hyperspace. An event is labelled a posteriori, with no indications allowing to attribute such event to a subset of the vector components.

Examination of a set of rare events show that 'some' vector components slightly deviate from a normal distribution (higher central peak, side peaks), and/or that their distribution has changed shape. But it's extremely difficult to pull a dominant factor, let alone find a subset of components that would allow to limit the scope of investigation, or even create an event fingerprint.

Put it simply: given a normal multivariate distribution, what algorithm, or investigation approach, would you recommend to classify (future) rare events (say occurence of 1/10), provided one has an history of all previous events, and in that history, all rare events are labelled.

In your day to day work, such rare events could be those preceding or indicative of an imminent stock-price drop.

Note:

  • KMeans clusterisation, Random Forest model, Bayes where tried, but did not allow to make any breakthrough

  • The assumption is that the distribution is normal multivariate. It looks it does, all vector components have a Gaussian distribution and are uncorrelated. When rare events occur, some vectors do deviate from a Gaussian though.

EDIT: My question didn't have much success, so I'll try to reformulate in a different way. Say there's a trading desk, with 1000 brokers, and their operation are well approximated using a multivariate normal distribution. In that desk, there are 5 brokers who on top of their normal operations, decide to do something else:

  • each guy picks a random number, p, if p <= p0, he places a different-order, following a normal distribution different from the one he normally has. If p > p0, he uses the originally assigned distribution.

  • Someone at the risk dept. classifies each batch of a 100 orders coming from the trading desk, using an algorithm of his own.

If p0 is small (otherwise, that's easy), How would you detect who the 5 rogue-guys are, and how would you identify the abnormal orders, if you had all the time series of the trading desk orders, labelled as good vs. suspicious orders?

Improve this question
$\endgroup$
7
  • $\begingroup$ You could take the zscore (or some other score based on the probability according to your modeling assumptions) of each return for each path, and then take the product of all of those (or sum of logs, for numerics reasons), and rank them by this. This will give you the most and least likely paths, given the model. Does that help? $\endgroup$
    – will
    Commented Apr 7, 2019 at 8:17
  • 1
    $\begingroup$ I would like to offer an opinion to this question but I struggle to understand it. There seems to be some doubt as to whether you want to maintain the core assumption of a multivariate normal distribution or not, and I am confused by your scenario, what does "their operations" being described by a MVN actually mean. What are their operations? More specifically, why is your problem easy for large $p_0$ and difficult for small $p_0$? How did you come to this assessment? $\endgroup$
    – Attack68 ♦
    Commented Apr 7, 2019 at 17:51
  • $\begingroup$ You do have a point. Say I create a vector V(b0,...bn), where each bi represents the activity of a broker, along a variable of interest for the trading desk. On the core assumption, when i analyse the data for normal transactions (if I use the scenario above), vector components are well described by multivariate distributions. For the abnormal one, a portion of them show abnormalities with respect to their normal counter parts (bi/tri-modal, asymmetry etc...). Note that those extra features seems continuous, in the sense that all components show varying degrees of differences from their.. $\endgroup$
    – Alex
    Commented Apr 8, 2019 at 5:59
  • $\begingroup$ original curves. As per what operations mean, I tried to translate my original problem into the quantitative finance/trading desk, but that might not be fully adequate due to my lack of experience in this field. I was thinking of the time series of transaction volume per broker, is that inadequate, MVN wise? $\endgroup$
    – Alex
    Commented Apr 8, 2019 at 6:11
  • $\begingroup$ Regarding your other question, If p0 ~ 1, well, all 5-rogues guys transactions are by definition abnormal, i.e. they always choose a normal distribution that is different from their past history. The problem arises from the fact that p0 is small enough, it becomes very difficult to distinguish a normal transaction from an abnormal one. In particular, in the bimodal case, when the abnormality is still within reasonable zscore from the original curve, it is almost impossible to define a finger print, although it is easy to create functions that unequivocally tell when a large enough ... $\endgroup$
    – Alex
    Commented Apr 8, 2019 at 6:29

1 Answer 1

Reset to default
2
$\begingroup$

If you’re comfortable making the assumption of multivariate normality (I’m not sure that you are), then this seems like a perfect place to use Mahalanobis distance.

One of the first facts that statistics students learn is the “empirical rule”: that $\sim 68\%$ of the density is within one standard deviation of the mean and $\sim 95\%$ of the density is within two standard deviations of the mean.

In high dimensions, it gets more complicated, but we can do something similar. Instead of measuring how many standard deviations we are from the mean, we measure how many units of Mahalanobis distance we are from the mean. In some sense, standard deviation is the univariate special case of Mahalanobis distance.

For better or for worse, the number of units of Mahalanobis distance containing a proportion of the density depends on the dimension $d$. We can calculate the exact number from quantiles of a $\chi^2_d$ distribution and then taking the square root.

If you want to contain the central $95\%$ in one dimension, find the $0.95$ quantile of $\chi^2_1$ and take the square root. In R software:

sqrt(qchisq(0.95, 1)) # I get the expected ~1.96

You mentioned that rare events for your purposes happen $1$ out of $10$ times. If you want to know how many units of Mahalanobis distance you need to be to have such a rare event, determine your dimension, d, and consult a $\chi^2_d$ distribution:

sqrt(qchisq(0.9, d))

As with the univariate empirical rule, this can fail miserably when the (multivariate, not just marginal) Gaussian assumption is violated, so be careful, and don’t let a common misinterpretation of the central limit theorem mislead you into thinking that a large sample size saves you.

Mahalanobis Distance

$$D_{M}({\vec {x}})={\sqrt {({\vec {x}}-{\vec {\mu }})^{\mathsf {T}}\mathbf {S} ^{-1}({\vec {x}}-{\vec {\mu }})}}$$

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.