I have recently began work on some high frequency financial tick data. I have been told to 'normalize' the data as much as possible and run linear regressions through them. In fact, the data doesn't seem to be anymore linear after I did transformations on them (box-cox/log etc) I understand the linear regressions bit, but most financial data is non-normal anyway, so why bother normalizing?
$\begingroup$
$\endgroup$
7
-
1$\begingroup$ cause otherwise you can't compare it? $\endgroup$– LazyCatCommented Jul 29, 2014 at 13:57
-
2$\begingroup$ Normalization typically means subtract the mean and divide by the standard deviation. That transformation won't make non-normal data normal. $\endgroup$– JohnCommented Jul 29, 2014 at 14:33
-
$\begingroup$ sorry, what i meant by normalizing is to make the distribution look like a normal distribution as much as possible. hence the 'normalize' $\endgroup$– evilbiscuitCommented Jul 30, 2014 at 8:34
-
$\begingroup$ @John OP mentioned "log". I am guessing this means taking the log or the log return? $\endgroup$– BCLCCommented Jul 31, 2014 at 5:22
-
$\begingroup$ @BCLC Well you could use logs or log returns with economic and financial data. Depends on what you're doing. $\endgroup$– JohnCommented Jul 31, 2014 at 13:02
|
Show 2 more comments
2 Answers
$\begingroup$
$\endgroup$
Short answer:
It offers some degree -- and in many cases, a greater degree -- of comparability between two types of data (different assets, returns, etc.)
Long answer:
You may already know this, but keep in mind that "normalization" can mean different things (see this question). There are various methods and purposes for normalizing data (financial or otherwise) but keep things in perspective. Normalize when doing so would be helpful for what you're trying to accomplish, and use a normalization technique that is appropriate. Linear regression has limitations, taking the logarithm has limitations, and so on. It's great to have a big toolbox of different data transformations, but part of that is knowing what to use.
As an aside, you're right that empirically markets have not exhibited normal returns. In fact, Mandelbrot explains in this article that the Pareto distribution is more realistic. It was published in 1963, but more recently he talks in this book about how the data have continued to demonstrate this pattern. The point is that you may read or hear about normalization techniques that rest on assumptions like normality that may not always be suited to the problem at hand. At the risk of editorializing, the assumption of normality has been subtly embedded in a lot of financial research and it may sometimes be misleading, so make sure to check the assumptions underlying what you're being told.
That being said, just because such an assumption may fail to hold with generality (either theoretically or empirically) does not mean it isn't useful for effectively navigating some specific problem from a mathematical or computational perspective. For example, if you have a model that's super-expensive computationally, and you can speed it up a bunch by making some simplifying assumption without significantly altering the results, perhaps doing so is worthwhile. (As long as you rigorously check that the results do in fact remain within a reasonable degree of similarity.)
$\begingroup$
$\endgroup$
2
got an answer from one of my pals, thought it might be interesting to share it here. The reason why we often use the normal distribution is because the distribution will be stable regardless of the number of samples (central limit theorem).
Imagine you had a normal distribution after transforming x amount of samples, and across time, u get more variables and we will want them to stay in the 'normal' distribution shape by exploiting the central limit theorem.
However, if you have exponential/Pareto (or any other) distributions, the distribution will tend to a normal distribution once the sample size is large enough due to the central limit theorem. This way, we can have a consistent model regardless of the time/sample size.
Hope this helps, and if anyone have different ideas about this, please do comment here!
-
$\begingroup$ Watch out for the assumptions underlying the central limit theorems! (there are many CLT, the most well-known one the CLT for i.i.d. [non-degenerate] random variables) You will no longer be looking at the actual variables when applying the i.i.d. CLT, but at their mean (transformed, in fact). $\endgroup$ Commented Aug 4, 2014 at 21:52
-
$\begingroup$ This post contains the same common misconception about the central limit theorem that many people believe at some point (including me). The central limit theorem certainly does not say that the empirical distribution of data drawn from a non-normal population converges to a normal distribution. In fact, under mild assumptions, the Glivenko-Cantelli theorem says why such convergence will not happen. By GC, the convergence is to the population distribution. $\endgroup$– DaveCommented Dec 31, 2021 at 2:38