I've started reading up on stochastic volatility models and it seems very difficult to discern which ones are used in practice and which have been mostly left alone in theory. What are the popular models used in the industry for stochastic volatility when pricing options, for what type of options are they usually employed and how are they implemented? I assume one straight forward way would be to do a two factor Monte-Carlo simulation of the stock price and the volatility, are there better ways?
Let me venture a guess.
If I had to design a system from scratch, I would probably prefer GARCH processes to properly stochastic conditional volatility processes. The fact that one step ahead, the conditional volatility process is known makes filtering both trivial and faster. Moreover, this class of option pricing model affords me all the flexibility of continuous time models: (1) I can build a leverage effect into it, (2) I can have conditionally heavy tailed returns using, say, inverse gaussian innovations, (3) I can mimmick the variance risk premium of models such as Heston's (1993), or Bakshi, Cao and Chen (1997), etc. using a quadratic pricing kernel...
Finally, if the speed of the pricing function is an issue, I would choose a combination of GARCH and return processes which admits an exponentially affine conditional moment generating function -- because, then, I can do very much like Heston (1993) or Heston and Nandi (2000): there is a formula similar in spirit to Black-Scholes-Merton that I can comptue in quasi-closed form. It's no wonder to me why Steve Heston himself has been such an heavy contributor to the literature on GARCH option pricing models: they are absurdly convenient tools.
If you depart from BSM, it's got to be worth your time. The thing is that if you're smart about calibration (i.e., if you don't take BSM seriously), your BSM model is a tougher benchmark than you think. Christoffersen and Jacobs (2004) actually made that point in a ManSci paper: if you taylor your loss function to what you want to do (e.g., minimize hedging errors when your goal is hedging), it's hell of a lot harder than it looks to outperform BSM.
But, all of this depends on what you want to do with it and on what type of options we're talking about. I was presuming equity options on indexes, so European options, and I am presuming speed and simplicity is of the essence. In that case, SV and GARCH models that fall in the affine class admit a quasi-closed form pricing formula and it's considerably faster than a Monte Carlo simulation. On the other hand, for a given strike, you really just need to simulate once: for the longest maturity. All others can be valued using mean values taken on earlier cross-sections of sample paths. So, it might look slow, but you can knock a few stones with one simulations.
Note that some continuous time models that are very interesting admits pricing through an inverse Fourrier transform like the GARCH models I have in mind and like the Heston (1993) model: Bakshi, Cao and Chen (1997) allows it, and a paper by Christoffersen, Heston and Jacobs (2009) published in Management Science allows it. The cool thing about the CHJ(2009) model is that they use two sources of volatility so they can get slow long-run movements and rapid short-run movements in the same model. This CHJ(2009) two factor SV model can also be mimicked by a component GARCH model as in Christoffersen, Dorion, Jacobs and Wang (2010) or, more recently, Babaoglu, Christoffersen, Heston and Jacobs (2018). The idea of a component GARCH goes back to Engel and Lee (1993) who proposed a transformation of a GARCH(2,2) model so you can think in terms of long-run and short-run conditional volatility movements.
The bottom line is that depending on what you do and what resources you have, different models might be preferable. From what I have heard, a lot of people actually still use Black-Scholes -- they're just smart enough to make sound adjustments to factor in things like the term structure of interest rate, the smile and the term structure of the smile. And given the conversations I had with people in finance departments, central banks and private banks, simplicity and speed are very important, so departures probably focus on things that allow quasi-analytical formulas as I said.