Given $N$ assets, the Markowitz mean-variance model requires expected returns, expected variances and a $N \times N$ covariance matrix. The joint distribution is fully defined by these measures.

However I often read that assets are required to be normally distributed for consideration in the mean-variance model. While I understand that a normal joint distribution is fully defined by the statistics described above, I can't really see why normality is required.

Can't we simply assume that the distribution is fully described by $\mu$, $\sigma^2$ and $\Sigma$, and not necessarily imply normality? That is, an obvious drawback is not considering higher moments which influence assets, such as skewness and kurtosis, but why is normality an assumption?