Is there any literature on the maths behind the computation of the price of an accreting swaption in the LMM model (no monte carlo, closed formula or close enough...)?
As long as the accretion of nominal is the same on the fixed and on the floating leg the Rebonato approximation for the equivalent Black volatility still works but the weights have to be multiplied (numerator and denominator) by the time dependent nominals. This is because the forward swap rate is still a weighted sum of forward Libors, albeit with the weights multiplied by the nominal.
From a practitioner standpoint, we know the prices of non accreting swaptions. The price of the accreting swaption in any model calibrated to these non accreting swaptions, is heavily dependent on the intra curve correlation assumptions in the model. We check that these correlations are consistent with other correlation dependent markets such as curve options.