1
$\begingroup$

I don't work in finance, but I've stumbled upon a problem that you guys may have to deal with in your jobs.

My problem is a random walk in high dim spaces ( > 100), in which I'm looking for vectors that explore a limited portion of that hyperspace. An event is labelled a posteriori, with no indications allowing to attribute such event to a subset of the vector components.

Examination of a set of rare events show that 'some' vector components slightly deviate from a normal distribution (higher central peak, side peaks), and/or that their distribution has changed shape. But it's extremely difficult to pull a dominant factor, let alone find a subset of components that would allow to limit the scope of investigation, or even create an event fingerprint.

Put it simply: given a normal multivariate distribution, what algorithm, or investigation approach, would you recommend to classify (future) rare events (say occurence of 1/10), provided one has an history of all previous events, and in that history, all rare events are labelled.

In your day to day work, such rare events could be those preceding or indicative of an imminent stock-price drop.

Note:

  • KMeans clusterisation, Random Forest model, Bayes where tried, but did not allow to make any breakthrough

  • The assumption is that the distribution is normal multivariate. It looks it does, all vector components have a Gaussian distribution and are uncorrelated. When rare events occur, some vectors do deviate from a Gaussian though.

EDIT: My question didn't have much success, so I'll try to reformulate in a different way. Say there's a trading desk, with 1000 brokers, and their operation are well approximated using a multivariate normal distribution. In that desk, there are 5 brokers who on top of their normal operations, decide to do something else:

  • each guy picks a random number, p, if p <= p0, he places a different-order, following a normal distribution different from the one he normally has. If p > p0, he uses the originally assigned distribution.

  • Someone at the risk dept. classifies each batch of a 100 orders coming from the trading desk, using an algorithm of his own.

If p0 is small (otherwise, that's easy), How would you detect who the 5 rogue-guys are, and how would you identify the abnormal orders, if you had all the time series of the trading desk orders, labelled as good vs. suspicious orders?

$\endgroup$
  • $\begingroup$ You could take the zscore (or some other score based on the probability according to your modeling assumptions) of each return for each path, and then take the product of all of those (or sum of logs, for numerics reasons), and rank them by this. This will give you the most and least likely paths, given the model. Does that help? $\endgroup$ – will Apr 7 at 8:17
  • 1
    $\begingroup$ I would like to offer an opinion to this question but I struggle to understand it. There seems to be some doubt as to whether you want to maintain the core assumption of a multivariate normal distribution or not, and I am confused by your scenario, what does "their operations" being described by a MVN actually mean. What are their operations? More specifically, why is your problem easy for large $p_0$ and difficult for small $p_0$? How did you come to this assessment? $\endgroup$ – Attack68 Apr 7 at 17:51
  • $\begingroup$ You do have a point. Say I create a vector V(b0,...bn), where each bi represents the activity of a broker, along a variable of interest for the trading desk. On the core assumption, when i analyse the data for normal transactions (if I use the scenario above), vector components are well described by multivariate distributions. For the abnormal one, a portion of them show abnormalities with respect to their normal counter parts (bi/tri-modal, asymmetry etc...). Note that those extra features seems continuous, in the sense that all components show varying degrees of differences from their.. $\endgroup$ – Alex Apr 8 at 5:59
  • $\begingroup$ original curves. As per what operations mean, I tried to translate my original problem into the quantitative finance/trading desk, but that might not be fully adequate due to my lack of experience in this field. I was thinking of the time series of transaction volume per broker, is that inadequate, MVN wise? $\endgroup$ – Alex Apr 8 at 6:11
  • $\begingroup$ Regarding your other question, If p0 ~ 1, well, all 5-rogues guys transactions are by definition abnormal, i.e. they always choose a normal distribution that is different from their past history. The problem arises from the fact that p0 is small enough, it becomes very difficult to distinguish a normal transaction from an abnormal one. In particular, in the bimodal case, when the abnormality is still within reasonable zscore from the original curve, it is almost impossible to define a finger print, although it is easy to create functions that unequivocally tell when a large enough ... $\endgroup$ – Alex Apr 8 at 6:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.