I used many statistical tools, i.e. t-SNE, SVM, Neural Network, UMAP, PCA, with the reformatted full market depth data with timestamp each second. UMAP gave me the best data representation, but clearly, it was just an illusion.

Cloud 1

Cloud 2

The two above pictures showed me an illusion. I thought I might see two distinct clusters stick together. Be aware that each color is a pre-defined label. The green points are just label 1 meaning the price is going up and the red color are just the label -1 meaning the price is going down. There's also the label 0 meaning the price is stable, but I did not consider here

If I am going closer, I see the following picture

Cloud 3

I analysed the data statistically and I realized there was just no trend.

I got the most valuable data and I have never seen trend analytically. How edge fund firm succeeded to find anomalies in the data so that they can apply market marking and make profit from them?

  • $\begingroup$ What is wrong with that question? I think it is a perfectly legitimate question. $\endgroup$
    – davegaut
    Dec 11 '18 at 23:14
  • 2
    $\begingroup$ I see you put effort in the question but unfortunately it is not clear to me what the concrete problem is that you want to see solved. If the question boils down too: How do I find a profitable trading strategy (like hedge funds do) the question is too broad. $\endgroup$
    – Bob Jansen
    Dec 12 '18 at 9:00

Your question is too broad to give anything but a very general answer. Data mining in the raw form won't do any good. At the minimum, you will pick up thousands of spurious correlations. You cannot go from data to a solution. You have to work in the opposite direction, you have to posit some model of the world and then test it. You must have an existing understanding of the relationships among the variables.

For example, let us imagine you believed there was a positive relationship between mentions on Twitter and volume traded. That may be true, but there are many lurking variables. Imagine two companies with equal capital bases and revenues, but with one difference. The first firm had one hundred distinct shareholders and the second firm had 5000 distinct shareholders. I say distinct because you want to ignore husband and wife, family office and related corporation combinations.

In the first company most shareholders, for whatever reason, are buy and hold shareholders, while the second trade quite a bit. If the second company had more comments on Twitter, well, that is unsurprising because they have more distinct shareholders. If the Twitter feed is a function of the number of stakeholders, then no new information about the trading pattern of the firm was added by using Twitter. Indeed, using Twitter may, in fact, reduce the information available as it is derived from the same dataset. It may be that counting shareholders is a superior metric than counting Twitter comments. Using both may trigger multicollinearity problems or trigger problems with the definition of the function. The Twitter comments may be a highly non-linear function of the count of shareholders.

You should never start in the data. You should start with your knowledge of the world, then use the data to disconfirm it.

Spurious relationships abound because accidental regularities abound. Think about the regularities in your life that look like they are not independent decisions, such as when you eat lunch. Notice that millions of people eat lunch at the same time as you. They do not have to and it isn't an intrinsic relationship. You could wake up an hour earlier, go to work before most people, and eat at lunch at ten in the morning. Those social accidents make it onto the tape.

  • 1
    $\begingroup$ Excellent answer. $\endgroup$
    – Bob Jansen
    Dec 12 '18 at 11:21

Not the answer you're looking for? Browse other questions tagged or ask your own question.