What is the difference between Smile Strangle margin and Market Strangle Margin in fx derivative market? Is it just variation in convention or is there any mathematical relationship between the two?
Do you mean actual margins for listed products or clearing on say CME? I am not familiar enough with this but a quick research shows that the CME seems to simply use 3 calls with a +1:-2:+1 configuration. Strikes need to be equidistant in this definition;
strike2 – strike1 = strike3 – strike2
Maybe I am wrong but I am assuming you simply mean smile butterflies and broker flys (also called market strangles). Generally, it's mainly a means of quoting FX vol.
ATM Delta neutral Straddles (DNS), Risk Reversals (RR) and Butterfly (BF) quotes completely describe the entire surface. ATM the level, RR the skew and BF the kurtosis. There are some complications like premium included / excluded Delta, Delta style (spot vs forward), spread models used etc.
One such complication involves how the BF is quoted. To get actual vols for calls and puts, one needs to transform these and solve for call / put vol.
Smile BF is simple:
RR = Vol of an OTM Call Option (C) - Vol of an OTM Put Option (P)
BF = ( C + P ) / 2 - Vol of ATM DNS
it's simple to show that
C = ATM + BF + RR/2
P = ATM + BF - RR/2
where you get say a 25 Delta call if you use 25D RR and BF respectively.
The above uses smile BF but brokers frequently quote market BF (short broker fly). These are a bit more involved but FX Volatility smile construction by Dimitri Reiswich and Uwe Wystup explains this in more detail (if this link does not work anymore, it can be searched easily).