0
$\begingroup$

In OLS regression, we have the normality of the error terms

$$\varepsilon \sim N(0,\sigma^2I_n)$$

I understand that we want to have a constant variance for homoscedastic errors, but why is $\sigma^2$ multiplied with the identity matrix ($I_n$)? Is it just in order to transform $\sigma^2$ from a scalar into a matrix?

$\endgroup$
3
  • 3
    $\begingroup$ It is a neat way of succinctly saying the following: 1. The error is a multivariate random vector with $n$ components (in this case 1 component is one observation) in the regression model. 2. The error has constant variance $\sigma^2$ in each component/observation because the variance of components correspond to the diagonal entries at each point in the covariance matrix. 3. The error components are uncorrelated/orthogonal, because covariances/correlations between different observations correspond to off-diagonal entries in the covariance matrix - which are all 0 for multiples of the identity. $\endgroup$ Sep 27 '20 at 16:51
  • 1
    $\begingroup$ @rubikscube09 Could you make this an answer? $\endgroup$
    – Bob Jansen
    Sep 27 '20 at 17:11
  • 1
    $\begingroup$ It is a widely-repeated myth that OLS requires normality of errors though that is not true. We only assume normality for hypothesis testing -- such as deciding if coefficient estimates are significant. Use the bootstrap and you need not even make that assumption. $\endgroup$
    – kurtosis
    Sep 28 '20 at 17:27
3
$\begingroup$

It is a neat way of succinctly saying the following:

  1. The error is a multivariate normal random vector with n components (in this case each component is one observation in the regression model.) That is to say: $$ \varepsilon = (\epsilon_1, \cdots, \epsilon_n) $$ where here each $\epsilon_i$ is a random variable that takes values in $\mathbb{R}$.
  2. The error has constant variance $\sigma^2$ in each component/observation because the variance of components correspond to the diagonal entries at each point in the covariance matrix. That is to say: $$ \mathrm{var}(\epsilon_i) = \sigma^2 $$ Moreover, the expectation of the errors is $0$.
  3. The error components are uncorrelated/orthogonal because covariances/correlations between different observations correspond to off-diagonal entries in the covariance matrix - which are all 0 for multiples of the identity. That is to say, the matrix with real entries: $$ \mathbb{E}[\epsilon \epsilon^T]_{i,j} = \mathrm{Cov}(\epsilon_i,\epsilon_j) = \begin{cases} = \sigma^2 & i = j \\ 0 & i \neq j\end{cases} $$ In particular: $$ \mathrm{cov}(\epsilon_i,\epsilon_j) = \mathrm{corr}(\epsilon_i,\epsilon_j) = 0 $$ for $i \neq j$, whereas: $$ \mathrm{cov}(\epsilon_i,\epsilon_i) = \mathrm{var}(\epsilon_i) = \sigma^2 $$ Because the errors have a multivariate normal distribution (they are an affine transformation of a vector whose components are independent normals), it follows that $\epsilon_i$ is in fact independent of $\epsilon_j$ (and not just uncorrelated).

All this said - putting the errors like this is a succinct notational way of summarizing the OLS assumption - $n$ error terms, all uncorrelated with each other, and all having constant variance.

$\endgroup$
1
  • 1
    $\begingroup$ Awesome answer, thanks a lot! $\endgroup$
    – Phil
    Sep 27 '20 at 19:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.