The idea is pretty much the same as the one used in the Breeden-Litzenberger result. You'll find many questions related to this here already, see e.g.: Prove that the butterfly condition is always greater than zero.
The current value of the derivative is the discounted expected value.
\begin{equation}
D_0 = e^{-r T} \int_K^\infty (x - K)^2 f(x) \mathrm{d}x,
\end{equation}
where $f$ is the risk-neutral probability density function of $S_T$. If you now carefully differentiate three times w.r.t. $K$ using the Leibniz rule, you get
\begin{eqnarray}
\frac{\partial D_0}{\partial K} & = & -2 e^{-r T} \int_K^\infty (x - K) f(x) \mathrm{d}x,\\
\frac{\partial^2 D_0}{\partial K^2} & = & 2 e^{-r T} \int_K^\infty f(x) \mathrm{d}x\\
\frac{\partial^3 D_0}{\partial K^3} & = & -2 e^{-r T} f(K).
\end{eqnarray}
Here, $f(K)$ has to be non-negative to be a valid probability density function and we thus conclude that the third derivative has to be non-positive. Note that this is either a typo in your question or the book.
Another way to obtain this result is to consider the finite difference approximation
\begin{equation}
\frac{\partial^3 D_0}{\partial K^3} \approx \frac{1}{\Delta^3} \left( -\frac{1}{2} D_0(K - 2\Delta) + D_0(K - \Delta) - D_0(K + \Delta) + \frac{1}{2} D_0(K + 2 \Delta) \right)
\end{equation}
for some step size $\Delta$. The terminal payoff of the portfolio $\Pi$ consisting of the option positions in the numerator is
\begin{eqnarray}
\Pi_T \left( S_T \right) & = & -\frac{1}{2} \left( S_T - K + 2 \Delta \right)^2 \mathrm{1} \left\{ S_T > K - 2 \Delta \right\} + \left( S_T - K + \Delta \right)^2 \mathrm{1} \left\{ S_T > K - \Delta \right\}\\
& & - \left( S_T - K - \Delta \right)^2 \mathrm{1} \left\{ S_T > K + \Delta \right\} + \frac{1}{2} \left( S_T - K - 2 \Delta \right)^2 \mathrm{1} \left\{ S_T > K + 2 \Delta \right\}.
\end{eqnarray}
We first observe that for any $x \geq 0$, we have
\begin{equation}
\Pi_T(K + x) = \Pi_T(K - x).
\end{equation}
We can thus restrict ourselves to analyzing the payoff for $S_T \leq K$ in the different intervals.
- For $S_T \leq K - 2 \Delta$, all options expiry out-of-the-money and $\Pi_T = 0$.
- For $K - 2 \Delta < S_T \leq K - \Delta$, only the short position in the strike $K - 2 \Delta$ is in-the-money and our payoff is strictly negative. At $S_T = K - \Delta$ it is $\Pi_T =
-\Delta^2 / 2$.
- For $K - \Delta < S_T \leq K$, also the long position in the strike $K - \Delta$ is in-the-money. However, our overall payoff becomes even more negative. At $S_T = K$ it is $\Pi_T = -\Delta^2$.
Using the symmetry around $S_T = K$, we find that our payoff is non-positive everywhere. Consequently, the initial portfolio value has to be non-positive as well and we conclude that $\partial^3 D_0 / \partial K^3 \leq 0$.