1
$\begingroup$

I'd appreciate you helping me understanding the proof of minimizing the sum of squared errors in linear regression models using matrix notation. I'm trying to derive enter image description here by minimizing the sum of squared errors,

enter image description here

Look at this proof,

The q.c.e. basic equation in matrix form is: 
  y = Xb + e
  where y (dependent variable) is (nx1) or  (5x1)
        X (independent vars) is (nxk) or  (5x3)
        b (betas) is (kx1) or  (3x1)
        e (errors) is (nx1) or  (5x1)
Minimizing sum or squared errors using calculus results in the OLS eqn:
  b=(X'X)-1.X'y
To minimize the sum of squared errors of
a k dimensional line that describes the relationship 
between the k independent variables and y we
find the set of slopes (betas) that minimizes
Σ_{i=1 to n} e_i^2
Re-written in linear algebra we seek to min e'e
Rearranging the regression model equation, we get e = y - Xb
So e'e = (y-Xb)'(y-Xb) = y'y - 2b'X'y + b'X'Xb   (see Judge et al (1985) p14 )
Differentiating by b we get 0 = - 2X'y + X'Xb -> 2X'Xb=2X'y
Rearranging, dividing both sides by 2 -> b = X'X-1X'y

it is stated that if you rewrite the summation expression above, it is

enter image description here

Is any summation over a squared vector/matrix the product of the transpose with the vector/matrix itself?

Further in the proof, e is substituted by Y-X*beta, and e'e is differentiated with respect to beta, yielding this expression

enter image description here

and it is stated that this is equal to

enter image description here

Does anybody know where the second 2 comes from? If I rearrange it, it yields X'X*beta = 2X'Y without the 2 in front of the lefthand side of the equation.

Thank you!

Improve this question
$\endgroup$
3
  • $\begingroup$ (1) You know how when you differentiate $x^2$ "a two comes out in front" and you get $2x$? The same happens in matrix calculus so when you differentiate b'X'Xb for b, you get 2X'Xb. They left that 2 out (typo), they should have written 0 = - 2X'y + 2X'Xb. Both twos should be there. $\endgroup$ – noob2 Sep 26 '20 at 10:02
  • $\begingroup$ (2) When $x$ and $y$ are vectors $x' y$ is the inner product of $x$ and $y$. And yes, the inner product of a vector with itself (i.e. $e' e$) is equal to the sum of squares of the elements. But if $x$ or $y$ are matrices, it is not the case. $\endgroup$ – noob2 Sep 26 '20 at 10:13
  • $\begingroup$ Thank you very much noob2! $\endgroup$ – Phil Sep 26 '20 at 10:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.