I am reading Prado's new book, Machine Learning for Asset Managers.

In the page1 of his book, there is this sentence.

To a greater extent than other mathematical disciplines, statistics is a product of its time. If Francis Galton, Karl Pearson, Ronald Fisher, and Jerzy Neyman had had access to computers, they may have created an entirely different field. Classical statistics relies on simplistic assumptions (linearity, independence), in-sample analysis, analytical solutions, and asymptotic properties partly because its founders had access to limited computing power.

I guess the independence in the quote originated from IID (independent, identical distribution) assumption.

But I'm not quite sure where the linearity of classical statistics comes from. The one place I could think of is about the linear regression. But it is pretty easy to extend linear regression to higher-order using the same Gauss-Markov theorem with the basis function approach.

  • $\begingroup$ @noob2 For me, classical statistics is inherinatly non-linear. Normal distribution is non-linear. They even talks of moment generating functions which is very non-linear. For the most of probability distribution, I don't find any linear theory at all. Have you heard of any linear algebra in statistics? Classifying classical statistics as linear is too much narrow view, I feel in the way. $\endgroup$ Apr 11 '20 at 14:34

It appears that you are using "linearity" in a litteral sense while De Prado is using it in a broader sense, which is quite common in Statistics.

In Statistics Linearity is not what the formula looks like, it is the properties and assumptions of the system under study. You consider the Normal distribution non-linear because it has an exponential and some squares in the formula. The statisticians frequently use the Normal distribution because it has the nice property that linear combinations of Normal variables are also Normal, which is not necessarily true for other distributions. The whole field of Kalman Filtering for example relies on this property and Non Linear and/or Non Gaussian filtering is very hard to do because you can no longer rely on this basic fact.

An important result of Classical Statistics is the Gauss Markov Theorem: Ordinary Least Squares provides the Best Linear Unbiased Estimator if the errors are (linearly) uncorrelated with mean zero and homoscedastic, finite, variance. Note the word linear appears twice.

Pearson Correlation is also called Linear Correlation, even though the formula has some squared terms in it. It is properties like $\rho(A+B,C)=\rho(A,C)+\rho(B,C)$ which make it linear.

As to the relationship between Classical Statistics and Linear Algebra, it is extensive. Consider for example that the linear regression can be written as $b=(X^T X)^{-1} X^T Y$. Yes, that involves Linear Algebra, it can be interpreted as a projection in a high dimensional linear space.

Like it or not the observations of De Prado you quoted are commonplace, he has not said anything original here (after all, he is still on Page 1 ;) ).

  • $\begingroup$ I agree that there is linearity in classical statistics but then, ML also assumes linearity here, and there such as the neurons in NN are combined linearly to give out the result. I don't agree with him putting the linearity of classical statistics with equal status as IID. LInearity in classical statistics doesn't play a decisive role as IID, in my point of view. And I don't think that the linearity is an important factor distinguishing classical statistics and ML either. $\endgroup$ Jun 15 '20 at 12:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.