I am not entirely sure what you are asking. I cannot answer from a valuation of contract perspective but I can help on an asset pricing perspective. Vol-of-vol is important to model a volatility swap because absent vol-of-vol (or jumps in the underlying) the variance risk premium on the contract is zero.
There are two relevant references:
- First Carr and Wu (2009) show that there is a large variance risk premium. Take a look at table 3 from their paper:
- Second vol-of-vol is important in generating this premium. Take a look at Bollerslev, Tauchen and Zhou (2009). They basically embed vol-of-vol in a standard asset pricing framework:
\begin{equation*}
\Delta c_{t+1} \equiv g_{t+1} = \mu_c +x_t + \sigma_t \eta_{t+1}
\end{equation*}
\begin{equation*}
\sigma_{t+1}^2 = a_\sigma + \rho_\sigma \sigma_t^2 + \sqrt{q_t} z_{\sigma,t+1}
\end{equation*}
\begin{equation*}
q_{t+1} = a_q + \rho_q q_t + \varphi_q \sqrt{q_t} z_{q,t+1}
\end{equation*}
The last equation is the vol-of-vol. They show that in such a framework the variance risk premium is given by:
\begin{eqnarray*}
E_t^Q\left(\sigma_{r, t+1}^2\right) - E_t\left(\sigma_{r, t+1}^2\right) = (\theta - 1) k_1^2 (A_\sigma^2 + A_q^2 \varphi_q^2) \varphi_q^2 ]q_t
\end{eqnarray*}
So you can immediatly see that if there is no stochastic vol-of-vol the variance risk premium is zero. Also if $q_t$ is constant the variance risk premium is not time-varying.