# Derivation of HJB equation

I am trying to derive the HJB equation in a stochastic setting. Let me exemplify my problem with the simplest case where there is no control, just one state variable. Assume the payoff is given by $$V(X_{t})\equiv E_{t}\left\{ \int_{t}^{\infty}e^{-\rho(s-t)}u(X_{s})ds\right\}$$ where $X_{t}$ is given by $$dX_{t}=\mu(X_{t})dt+\sigma(X_{t})dZ_{t}$$ and $Z_{t}$ is the standard Brownian Motion. For any $dt>0$ we can write: $$V(X_{t})=E_{t}\left\{ \int_{t}^{t+dt}e^{-\rho(s-t)}u(X_{s})ds+e^{-\rho dt}V(X_{t+dt})\right\}$$

$$\left(1-e^{-\rho dt}\right)V(X_{t})=E_{t}\left\{ \int_{t}^{t+dt}e^{-\rho(s-t)}u(X_{s})ds+e^{-\rho dt}\left[V(X_{t+dt})-V(X_{t})\right]\right\} \tag{1}$$

From Ito calculus we get that (and assuming that $W(\cdot)$ is well behaved): $$V(X_{t+dt})-V(X_{t})=\int_{t}^{t+dt}V'(X_{s})dX_{s}+\frac{1}{2}\int_{t}^{t+dt}V''(X_{s})d[X_{s}]=\int_{t}^{t+dt}V'(X_{s})dX_{s}+\frac{1}{2}\int_{t}^{t+dt}\sigma(X_{s})^2V''(X_{s})ds$$ where the last equality follows from the known properties of the quadratic variation of the process $X_{t}$. Plugging this back in (1): $$\left(1-e^{-\rho dt}\right)V(X_{t})=E_{t}\left\{ \int_{t}^{t+dt}e^{-\rho(s-t)}u(X_{s})ds+e^{-\rho dt}\left[\int_{t}^{t+dt}V'(X_{s})dX_{s}+\frac{1}{2}\int_{t}^{t+dt}\sigma(X_{t})^2V''(X_{s})ds\right]\right\}$$

Dividing both sides by $dt$ and taking the limit $dt\rightarrow0$: $$\rho V(X_{t})=E_{t}\left\{ u(X_{t})+\lim_{dt\rightarrow0}\frac{\int_{t}^{t+dt}V'(X_{s})dX_{s}}{dt}+\frac{1}{2}\sigma(X_{t})^2V''(X_{t})\right\}$$ where I used the fact that when dealing with the Riemann integral: $\lim_{dt\rightarrow0}\frac{\int_{t}^{t+dt}f(x_{s})ds}{dt}=f(x_{t})$ (from standard calculus).

As you can see, I am almost there. I just don't know how to deal with term $\lim_{dt\rightarrow0}\frac{\int_{t}^{t+dt}V'(X_{s})dX_{s}}{dt}$. For example, assume $\mu(X_{t})=0$ and $\sigma(X_{t})=1$, so that $X_{t}$ is simply the standard Brownian Motion $Z_{t}$. In that case, to get the HJB formula right I would need: $$\lim_{dt\rightarrow0}\frac{\int_{t}^{t+dt}V'(Z_{s})dZ_{s}}{dt}=0$$ But I don't know how to prove that this is true. More generally (for any $\mu(X_{t})$ and $\sigma(X_{t})$), I would need to prove: $$\lim_{dt\rightarrow0}\frac{\int_{t}^{t+dt}V'(X_{s})dX_{s}}{dt}=\mu(X_{t})V'(X_{t})$$ which I am also not sure how to do. Any ideas?

• I am not sure to understand the point of the deriving the equation if there is no control, Also good notations help. Writing $W$ for you payoff is not a good choice. And your payoff is a function of time and state so $v(t,x)$ would be much better than $W(X_t)$. To answer your question simply replace $dX_t = \mu dt + \sigma dZ_t$, the expected value of the brownian part is $0$. PS: you forgot to square the volatility in the quadratic term. – AFK May 21 '15 at 19:32
• I changed the notation. I also removed the dependence of the drift and variance on time, so that I don't need to make my value function depend on time. In some economics applications it is useful to write the value function as a differential equation. Even if it is only to get some economic intuition from HJB. – Pcw. May 21 '15 at 19:59
• You need only note that \begin{align*} E_{t}\bigg( \lim_{dt\rightarrow0}\frac{\int_{t}^{t+dt}V'(X_{s})dX_{s}}{dt}\bigg) &= \lim_{dt\rightarrow0}E_{t}\bigg(\frac{\int_{t}^{t+dt}V'(X_{s})dX_{s}}{dt}\bigg)\\ &=\lim_{dt\rightarrow0}E_{t}\bigg(\frac{\int_{t}^{t+dt}V'(X_{s})u(X_{s})dt + \int_{t}^{t+dt}V'(X_{s})\sigma(X_{s})dZ_s}{dt}\bigg)\\ &= \lim_{dt\rightarrow0}E_{t}\bigg(\frac{\int_{t}^{t+dt}V'(X_{s})u(X_{s})dt }{dt}\bigg)\\ &=V'(X_{t})u(X_{t}), \end{align*} as pointed out by AFK. – Gordon May 21 '15 at 20:26

By using the fact that the brownian integral has expected value $0$, we find \begin{eqnarray*} & & \left(1-e^{-\rho dt}\right)v(X_{t}) \\ &=& E_{t}\left\{ \int_{t}^{t+dt}e^{-\rho(s-t)}u(X_{s})ds+e^{-\rho dt}\left[\int_{t}^{t+dt}v'(X_{s})dX_{s}+\frac{1}{2}\int_{t}^{t+dt}\sigma(X_{s},s)^2 v''(X_{s})ds\right]\right\} \\ &= & E_{t}\left\{ \int_{t}^{t+dt}e^{-\rho(s-t)}u(X_{s}) +e^{-\rho dt}\left(v'(X_{s})\mu(s,X_s) +\frac{1}{2}\sigma(X_{s},s)^2 v''(X_{s})\right)ds\right\} \end{eqnarray*} So the PDE is \begin{eqnarray*} \rho v(x) &=& u(x) + \left(\mu(t,x) v'(x) +\frac{1}{2}\sigma(t,x)^2 v''(x)\right) \end{eqnarray*} or \begin{eqnarray*} (- \mathcal{A}_t + \rho) v(x) &=& u(x) \end{eqnarray*} where $\mathcal{A}_t = \mu(t,x)\partial_x + \frac{1}{2}\sigma(t,x)^2 \partial_x^2$ is the generator of the diffusion.
• You are saying that $E_t\{\int_t^{t+dt} v'(x_s)dZ_s\}=0$? I've just started to deal with continuous time stuff, but it is not obvious to to me why (I get that $E_t\{\int_t^{t+dt}dZ_s\}=0$ though). – Pcw. May 21 '15 at 20:38
• You need integrability conditions on $v^{'}$. It must be adaptable and belong to ${{\cal L}^2}$ then you have an Ito integral whose expectation must be zero. – Rohit Arora Jun 21 '15 at 18:30