Derivation of HJB equation

Question

I am trying to derive the HJB equation in a stochastic setting. Let me exemplify my problem with the simplest case where there is no control, just one state variable. Assume the payoff is given by $$ V(X_{t})\equiv E_{t}\left\{ \int_{t}^{\infty}e^{-\rho(s-t)}u(X_{s})ds\right\} $$ where $X_{t}$ is given by $$ dX_{t}=\mu(X_{t})dt+\sigma(X_{t})dZ_{t} $$ and $Z_{t}$ is the standard Brownian Motion. For any $dt>0$ we can write: $$ V(X_{t})=E_{t}\left\{ \int_{t}^{t+dt}e^{-\rho(s-t)}u(X_{s})ds+e^{-\rho dt}V(X_{t+dt})\right\} $$

$$ \left(1-e^{-\rho dt}\right)V(X_{t})=E_{t}\left\{ \int_{t}^{t+dt}e^{-\rho(s-t)}u(X_{s})ds+e^{-\rho dt}\left[V(X_{t+dt})-V(X_{t})\right]\right\} \tag{1} $$

From Ito calculus we get that (and assuming that $W(\cdot)$ is well behaved): $$ V(X_{t+dt})-V(X_{t})=\int_{t}^{t+dt}V'(X_{s})dX_{s}+\frac{1}{2}\int_{t}^{t+dt}V''(X_{s})d[X_{s}]=\int_{t}^{t+dt}V'(X_{s})dX_{s}+\frac{1}{2}\int_{t}^{t+dt}\sigma(X_{s})^2V''(X_{s})ds $$ where the last equality follows from the known properties of the quadratic variation of the process $X_{t}$. Plugging this back in (1): $$ \left(1-e^{-\rho dt}\right)V(X_{t})=E_{t}\left\{ \int_{t}^{t+dt}e^{-\rho(s-t)}u(X_{s})ds+e^{-\rho dt}\left[\int_{t}^{t+dt}V'(X_{s})dX_{s}+\frac{1}{2}\int_{t}^{t+dt}\sigma(X_{t})^2V''(X_{s})ds\right]\right\} $$

Dividing both sides by $dt$ and taking the limit $dt\rightarrow0$: $$ \rho V(X_{t})=E_{t}\left\{ u(X_{t})+\lim_{dt\rightarrow0}\frac{\int_{t}^{t+dt}V'(X_{s})dX_{s}}{dt}+\frac{1}{2}\sigma(X_{t})^2V''(X_{t})\right\} $$ where I used the fact that when dealing with the Riemann integral: $\lim_{dt\rightarrow0}\frac{\int_{t}^{t+dt}f(x_{s})ds}{dt}=f(x_{t})$ (from standard calculus).

As you can see, I am almost there. I just don't know how to deal with term $\lim_{dt\rightarrow0}\frac{\int_{t}^{t+dt}V'(X_{s})dX_{s}}{dt}$. For example, assume $\mu(X_{t})=0$ and $\sigma(X_{t})=1$, so that $X_{t}$ is simply the standard Brownian Motion $Z_{t}$. In that case, to get the HJB formula right I would need: $$ \lim_{dt\rightarrow0}\frac{\int_{t}^{t+dt}V'(Z_{s})dZ_{s}}{dt}=0 $$ But I don't know how to prove that this is true. More generally (for any $\mu(X_{t})$ and $\sigma(X_{t})$), I would need to prove: $$ \lim_{dt\rightarrow0}\frac{\int_{t}^{t+dt}V'(X_{s})dX_{s}}{dt}=\mu(X_{t})V'(X_{t}) $$ which I am also not sure how to do. Any ideas?

Answer 1

By using the fact that the brownian integral has expected value $0$, we find \begin{eqnarray*} & & \left(1-e^{-\rho dt}\right)v(X_{t}) \\ &=& E_{t}\left\{ \int_{t}^{t+dt}e^{-\rho(s-t)}u(X_{s})ds+e^{-\rho dt}\left[\int_{t}^{t+dt}v'(X_{s})dX_{s}+\frac{1}{2}\int_{t}^{t+dt}\sigma(X_{s},s)^2 v''(X_{s})ds\right]\right\} \\ &= & E_{t}\left\{ \int_{t}^{t+dt}e^{-\rho(s-t)}u(X_{s}) +e^{-\rho dt}\left(v'(X_{s})\mu(s,X_s) +\frac{1}{2}\sigma(X_{s},s)^2 v''(X_{s})\right)ds\right\} \end{eqnarray*} So the PDE is \begin{eqnarray*} \rho v(x) &=& u(x) + \left(\mu(t,x) v'(x) +\frac{1}{2}\sigma(t,x)^2 v''(x)\right) \end{eqnarray*} or \begin{eqnarray*} (- \mathcal{A}_t + \rho) v(x) &=& u(x) \end{eqnarray*} where $\mathcal{A}_t = \mu(t,x)\partial_x + \frac{1}{2}\sigma(t,x)^2 \partial_x^2$ is the generator of the diffusion.