You can find the derivation of the Heston characteristic function (its Fourier Transform) in Gatheral (2006).
Using the characteristic function, you can optimize the model on the prices. There are multiple approaches to optimize, among others pattern search (which is very slow) and stochastic optimization (randomly jump around and stop after n iterations), but i recommend a mix of both. I often use adaptive simulated annealing for an inital calibration and then run a pattern search. Depending on the language you use, these are available as functions and its pretty simple to implement.
If I recall correctly, the Fourier transform/characteristic function of the Heston model is
$$ \phi_T(u) = \exp\{C(u,\tau)\theta + D(u,\tau)v_0\}$$
where
$$ C(u,\tau)=\ \kappa \left[r_{-} \tau - \frac{2}{\eta^2}\log\left(\frac{1-g e^{-d\tau}}{1-g}\right) \right] $$
$$D(u,\tau)=\ r_{-} \frac{1-e^{-d\tau}}{1-ge^{-d\tau}} $$
$$g =\ \frac{r_{-}}{r_{+}} $$
$$r_{\pm} =\ \frac{b\pm d}{\eta^2} $$
$$d =\ d=\sqrt{b^2-4ac} $$
$$c =\ \frac{\eta^2}{2} $$
$$b =\ \kappa-\rho\eta iu$$
$$a =\ -\frac{u^2}{2} - \frac{iu}{2} $$
Gatheral provides derivations for SVJ, SVJJ, VarG, etc as well.
