I understand the volatility surface for swaption is built using implied vols of ATM swaptions. I had a question on the instruments that are used. Should the instruments used change depending on the terms of the underlying swap? For example, if the underlying is a 6M LIBOR floating swap, then should we use instruments referencing 6M LIBOR to construct the vol surface? My understanding was that we will use the most liquid instruments which is swaps referencing 3M LIBOR.
Yes, in general, you should match the swaptions such that the underlying in the swaptions match what you are trying to calibrate. Consider, that swaption(lognormal vols) = Annuity * function(F, K, vol, time). The F, is the forward and it has to match the underlying instrument.
If you do not, you are capturing the dynamics of a different instruments. Unless, your analytics has the conversion mathematics to shift from one tenor (such as 3m-float) to another tenor (such as your 6mth-float). Even then, these kind of conversions usually require a convexity adjustment that takes in the covariance of the 3m-6mth tenor spreads.