Consider the process,
$$ dX_t=(-aX_t+b(1-X_t))dt + \sqrt{X_t(1-X_t)}dW_t $$
How do I show that the stationary distribution for the transition density is a beta distribution?
I tried expanding the corresponding Kolmogorov Forward Equation but it seems too difficult to solve the equation.
There is a shortcut around the Forward Equation when you are looking for the stationary distribution. Let me write $$ dX = \mu(X)dt +\sigma(X)dW $$ for $$ \mu(x)=b(1-x)-ax\ \text{ and }\ \sigma^2(x)=x(1-x) $$
The Forward Equation indeed states that the stationary distribution $p(x)$ will be satisfied for $\partial p/\partial t = 0$, therefore $$ \frac{1}{2}\frac{d^2}{dx^2}\left[\sigma^2(x)p(x)\right] - \frac{d}{dx}\left[\mu(x)p(x)\right] = 0 $$
The trick is to take one differential as a common factor and write $$ \frac{d}{dx} \left\{ \frac{1}{2}\frac{d}{dx}\left[\sigma^2(x)p(x)\right] - \mu(x)p(x) \right\}= 0 $$ Then, the term in the braces will be a constant (it's derivative is zero), and we can take it to be zero. Then we are facing the first order ODE $$ \frac{1}{2}\frac{d}{dx}\left[\sigma^2(x)p(x)\right] = \mu(x)p(x) $$ Solving this yields the stationary distribution up to the normalization constant. The solution is actually given by $$ p(x) \propto \sigma^{-2}(x) \exp\left( \int^x \frac{2\mu(u)}{\sigma^2(u)} du \right) $$
The above holds for any process. In your particular case, the integral becomes $$ \int^x \frac{2b(1-u)-2au}{u(1-u)} du = 2b \int^x \frac{du}{u} -2a\int^x \frac{du}{1-u} = \log \left( x^{2b} (1-x)^{2a} \right) $$
Hence, overall the stationary distribution is Beta with parameters $(\alpha,\beta)=(2b,2a)$ $$ p(x) \propto x^{2b-1} (1-x)^{2a-1} $$
try to calculate the moment generating function and show that it correspond to the one of a beta distributed random variable
