When ATM implied volatility is higher than OTM put and call I believe that the volatility smile is no longer arbitrage free? Why is that?
On the other hand, when ATM implied volatility is lower than OTM put and call is the volatility smile always arbitrage free? Why is that?
I generally agree with @dm63's answer: A convex (concave) smile around the forward usually indicates and leptokurtic (platykurtic) implied risk-neutral probability density. Both situations can or cannot admit arbitrage. I provide you with two counterexamples to your statements.
A volatility smile that is concave around the forward does not necessarily represent an arbitrage.
Concave smiles often arise when a significant jump with a predictable time of occurrence is priced in. This is often the case for single stocks around quarterly earnings announcements or for indices around macro events such as elections, referendums or rate decisions.
Consider for example an underlying asset that does not move except for by a single jump. Let $X_t = \ln \left( S_t / S_0 \right)$ and define
\begin{equation} X_t = \int_0^t \gamma(u) \mathrm{d}u + Y \mathrm{1} \left\{ t \geq t_J \right\}. \end{equation}
Here, the jump time $t_J$ is known and has the random jump size $Y$. $\gamma$ is a deterministic drift that is chosen such that the discounted asset prices is a martingale under the risk-neutral probability measure $\mathbb{P}^*$. It is given by
\begin{equation} \gamma(t) = r - \ln \left( \phi_Y(-\mathrm{i}) \right) \delta \left( t - t_J \right), \end{equation}
where $\phi_Y(\omega)$ is the characteristic function of $Y$ and $\delta$ is the Dirac delta function.
Assume that $Y$ follows a normal mixture distribution, i.e.
\begin{equation} Y \sim \begin{cases} Y_1 & \text{with probability }p\\ Y_2 & \text{with probability } 1 - p.\end{cases} \end{equation}
where $Y_1 \sim \mathcal{N} \left( \mu_1, \sigma_1^2 \right)$ and $Y_2 \sim \mathcal{N} \left( \mu_2, \sigma_2^2 \right)$. This model usually generates platykurtic implied densities and concave implied volatility smiles.
Here is a numerical example. Let $t_J = 1 \text{ day}$, $\mu_1 = -5\%$, $\mu_2 = +5\%$, $\sigma_1 = \sigma_2 = 2\%$ and $p = 50\%$. Further let $S_0 = 100$, $r = 0\%$ and consider a maturity of $T = 1 \text{ week}$. We get the following implied density and volatility smile.
In practice, you would consider more complex/realistic underlying dynamics such as e.g. a stochastic volatility and/or jump-diffusion model.
To give you a real-world example: Here is the DAX 30 implied volatility smile as of December 1, 2016 for the maturity December 9, 2016. There was a jump priced in on the night of Sunday December 5th due to the Italian referendum which roughly had the implied parameters $\mu_1 = +2\%$, $\mu_2 = -3.5\%$, $\sigma_1 = \sigma_2 = 1.5\%$ and $p = 70\%$.
A volatility smile that is convex around the forward is not necessarily arbitrage-free.
A few popular implied volatility smile parametrizations are not arbitrage-free for their full parameter range.
Roper (2010) for example shows that the so-called "arbitrage free" original SVI parametrization due to Gatheral (2004) is actually not arbitrage free, even for realistic parameter combinations; see Figures 1 and 2 in his paper.
Another example is the Hagan et al. (2002) SABR parametrization, which is known to generate negative densities on the far downside strikes.
For both examples, there exists a vast body of literature that aims at providing arbitrage-free alternative formulations.
References
Gatheral, Jim (2004) "A Parsimonious Arbitrage-Free Implied Volatility Parametrization", Presentation, Global Derivatives & Risk Management 2004
Hagan, Patrick S., Deep Kumar, Andrew S. Lesniewski and Diana E. Woodward (2002) "Managing Smile Risk", Wilmott Magazine
Roper, Michael (2010) "Arbitrage Free Implied Volatility Surfaces", Working Paper, University of Sydney
Neither situation is necessarily an arbitrage. Negative smile is consistent with a 'thin-tailed' density function , just as positive smile is consistent with a fat tailed density function . It's true that an extreme amount of negative smile could cause the implied density to be negative in places I.e an arbitrage.
