I don't work in finance, but I've stumbled upon a problem that you guys may have to deal with in your jobs.
My problem is a random walk in high dim spaces ( > 100), in which I'm looking for vectors that explore a limited portion of that hyperspace. An event is labelled a posteriori, with no indications allowing to attribute such event to a subset of the vector components.
Examination of a set of rare events show that 'some' vector components slightly deviate from a normal distribution (higher central peak, side peaks), and/or that their distribution has changed shape. But it's extremely difficult to pull a dominant factor, let alone find a subset of components that would allow to limit the scope of investigation, or even create an event fingerprint.
Put it simply: given a normal multivariate distribution, what algorithm, or investigation approach, would you recommend to classify (future) rare events (say occurence of 1/10), provided one has an history of all previous events, and in that history, all rare events are labelled.
In your day to day work, such rare events could be those preceding or indicative of an imminent stock-price drop.
KMeans clusterisation, Random Forest model, Bayes where tried, but did not allow to make any breakthrough
The assumption is that the distribution is normal multivariate. It looks it does, all vector components have a Gaussian distribution and are uncorrelated. When rare events occur, some vectors do deviate from a Gaussian though.
EDIT: My question didn't have much success, so I'll try to reformulate in a different way. Say there's a trading desk, with 1000 brokers, and their operation are well approximated using a multivariate normal distribution. In that desk, there are 5 brokers who on top of their normal operations, decide to do something else:
each guy picks a random number, p, if p <= p0, he places a different-order, following a normal distribution different from the one he normally has. If p > p0, he uses the originally assigned distribution.
Someone at the risk dept. classifies each batch of a 100 orders coming from the trading desk, using an algorithm of his own.
If p0 is small (otherwise, that's easy), How would you detect who the 5 rogue-guys are, and how would you identify the abnormal orders, if you had all the time series of the trading desk orders, labelled as good vs. suspicious orders?