The capital asset pricing model (CAPM) is based on mean-variance utility; investors choose their portfolio based only on its mean and variance.
This is an entirely different approach than expected utility maximisation.
You can express the expected utility using its Taylor approximation around the expected wealth, with $\mu = E[W], \Delta = W - \mu$ (note that $E[\Delta]=0$):
E[U(W)] &= E[U(\mu + \Delta)]
\\ &= E\left[U(\mu) + \Delta \cdot U' + \frac 1 2 \Delta^2 U'' + \frac 1 6 \Delta^3 U''' + ...\right]
\\ &= U(\mu) + \frac 1 2 \sigma^2 U''(\mu) + \frac 1 6 E[\Delta^3] U''' + ...
Now, to turn this into a mean-variance utility (ie only a function of $\mu, \sigma$), there are 3 ways:
Impose conditions on the utility function: Take an expected utility function $U$ whose higher derivatives (above second derivative) vanish. That is the idea with quadratic utility (which, as you point out, has many problems).
Impose conditions on the returns, ie the distribution of $W$. It can be shown that one can express the above as a function of only $\mu, \sigma$ for elliptical distributions, no matter what $U$ is chosen. Those can have a restricted domain (ie limited liability, though then also limited upside, if I'm not mistaken).
Impose joint conditions on both the expected utility function and the distribution of $W$. That's complicated.
So why do we assume quadratic utility?
Easy to handle analytically.
Are there no other simple, more realistic functional forms for utility that would still lead to a reasonably clean portfolio optimization theory?
Mean-variance utility has many problems. For example, a mean-variance investor could refuse a gift of a limited liability asset, if it's volatile enough. It doesn't make much sense.
Not aware of a clean portfolio optimisation theory under more realistic assumptions.
Or are the issues I cited about the quadratic just negligible in practice?
Oh no, they are real. But mean-variance analysis or CAPM generally are hardly used in practice, but rather as conceptual tools, I'd say.