The VIX as a clear definition as defined in this paper I am interested to know why they came up with this formula.
I smell some reasonably complicated explanation here so any pointer to a paper would be fine with me. Thanks
In that white paper itself they quote where it came from: “More than you ever wanted to know about volatility swaps” by Kresimir Demeterfi, Emanuel Derman, Michael Kamal and Joseph Zou, Goldman Sachs Quantitative Strategies Research Notes, March 1999. This is a classic article which you should definitely read if you are trading volatility. While there might be a clearer explanation somewhere, this is the original and quite comprehensive work.
For some intuitive ideas behind this, also see this post in OnlyVIX blog.
The calculation of VIX itself is taking this Volatility Swaps idea and approximating it with discrete set of options (sum instead of integral), throwing away some ridiculously OTM options (prices of which are not representative at all). Since there is not always an expiration 30 days ahead, they approximate this point by interpolating (or extrapolating). These are merely some technical things they have to do to apply the formula.