The Heston model is represented by the bivariate system of stochastic differential equations (SDE)
\begin{align}
& d{{S}_{t}}=rS_tdt+{\sqrt\upsilon_t} d{{W}_{1}}(t) \\
& d{{\upsilon}_{t}}=\kappa(\theta-\upsilon_t) dt+\sigma{\sqrt\upsilon_t}d{{W}_{2}}(t) \\
\end{align}
The most popular way to estimate the parameters of the Heston model is with loss
functions. This method uses the error between quoted market prices and model
prices, or between market and model implied volatilities. The parameter estimates
$\hat\Theta=(\hat\kappa, \hat\theta,\hat\sigma,\hat\upsilon_o,\hat\rho)$ are those values which minimize the value of the loss function, so that the model prices or implied volatilities are as close as possible to their market counterparts.
A constrained minimization algorithm must be used in this regard so that the
constraints on the parameters
\begin{align}
\kappa>0\ ,\ \theta>0\ , \ \sigma>0 \, \ ,\upsilon_0>0 \, \ , \,\rho\in[-1,1]
\end{align}
are respected. Since loss functions use market option prices (or implied volatility
derived from those prices) as inputs, they produce estimates of the risk-neutral
parameters of the Heston model.Suppose we have a set of $N_T $ maturities $\tau_i$($ i=1,2,...,N_T)$ and set $N_K$ strikes $K_k$ ($k=1,2,...,N_k$).For each maturity For each maturity-strike combination $(\tau_t,\ K_k)$ we have market price $P(\tau_t , K_k)$ and a corresponding model price $P(\tau_t , K_k;\Theta)=P_{t,k}^{\Theta}$ generated
by the Heston model. Attached to each option is an optional weight $w_{t,k}$ . There are
many possible ways to define a loss function, but they usually fall into one of two
categories: those based on prices, and those based on implied volatilities.
The first category of loss functions are those that minimize the error between
quoted and model prices. The error is usually defined as the squared difference
between the quoted and model prices, or the absolute value of the difference; relative
errors can also be used. For example, parameter estimates obtained using the mean
error sum of squares (MSE) loss function are obtained by minimizing
\begin{align}
\frac{1}{N}\sum_{t,k}w_{t,k}(P_{t,k}-P_{t,k}^{\Theta})^2
\end{align}
with respect to $\Theta$ where $N$ is the number of quotes. The relative mean error sum of
squares (RMSE) parameter estimates are obtained with the loss function
\begin{align}
\frac{1}{N}\sum_{t,k}w_{t,k}\frac{(P_{t,k}-P_{t,k}^{\Theta})^2}{P_{t,k}}
\end{align}
The second category of loss functions are those that minimize the error between
quoted and model implied volatilities. Again, the error is usually defined as the
squared difference, absolute difference, or relative difference, between quoted and
model implied volatilities. This category of loss function is sensible, since options
are often quoted in terms of implied volatility, and since the fit of model is often
assessed by comparing quoted and model implied volatilities. Hence, for example,
the implied volatility mean error sum of squares (IVMSE) parameter estimates are
based on the loss function
\begin{align}
\frac{1}{N}\sum_{t,k}w_{t,k}(IV_{t,k}-IV_{t,k}^{\Theta})^2
\end{align}
where $IV_{t,k}$ and $IV_{t,k}^\Theta$ are are the quoted and model implied
volatilities, respectively. The relative and absolute versions can also be used.Estimation of the Heston model parameters by loss functions has been used by
Bakshi, Cao, and Chen (1997), Bams et al. (2009), Christoffersen and Jacobs (2004),
Mikhailov and No¨ gel, (2003), and many others. There is no consensus on which
loss function is the best one, but Christoffersen and Jacobs (2004) point out that
the same loss function should be used for parameter estimation and for evaluating
model fit.