How do you actually solve a stochastic HJB equation in practice?

Question

I've read a number of recent papers on market making. Nearly all of the more recent papers focus on defining the problem in terms of a state and action space, deriving the relevant HJB equations and constraints, and going from there.

I realize this could be viewed as a broad question, but I think that it's worthy from the perspective of a programmer. In practice, what are the general methods to attack the problem of solving an optimal policy in real time? I understand the theoretical underpinnings of the DPP, but there are few libraries or frameworks designed to help with this, and those that exist are extremely light on the documentation. I'm confused as to how viable methods like policy and value iteration are when solving online (in terms of computational cost and resulting added latency in the HFT world). Are there any good examples of someone taking a set of HJB equations for optimal market making (inventory management) or execution and going all the way (preferably with code or pseudocode) to a working mvp?

Answer 1

An HJB is made of two components:

• a "core component" that corresponds to applying an optimal control
• the "dynamics" of the value function (that is surrounding this optimizattion).

Take the one dimension version of the HJB (2.3) of L, C. A., & Mouzouni, C. (2019). A mean field game of portfolio trading and its consequences on perceived correlations. that I know well: $${\gamma\over 2} q^2 =\partial_t u + a \mu\; \partial_s u+\sup_\nu \left\{ \nu\partial_q u - \ell \;\nu^2/V) \right\}$$ with the terminal condition $u_T=-Aq^2$ (here I took $L(x)=\ell x^2$ in the orginal equality to make it more simple).

• The part $\sup_\nu \left\{ \nu\partial_q u - \ell \;\nu^2/V) \right\}$ corresponds to applying an optimal control $\nu$ that is a speed of trading $\nu$. You may recognize the instantaneous market impact cost in $\nu^2$ and the "bleeding" part $\nu\partial_q u$ (it says that trading somehow corresponds to changing your inventory).

• The other terms are

  • the cost ${\gamma\over 2} q^2$ of keeping an open exposure
  • the natural decay of the value function with time $\partial_t u$
  • and the permanent market impact term $a \mu\; \partial_s u$ that changes the value of your position.

To solve it a simple way is to apply a sequence of value-policy iterations:

1. you can start with a arbirary (but not too stupid) guess for the value $u_n(t,q)$ to get a position of size $q$ at $t$.
2. thanks to that you can solve the optimal control part and find on optimal trading speed $\nu^* $ that solves the supremum (for each $(t,q)$). You can do it numerically in general but here it can be done in close-form: $\nu^*=\partial_q u_n\cdot V / (2\ell)$.
3. thanks to that you can plug it into the HJB and find a PDE. For us it will be $${\gamma\over 2} q^2 =\partial_t u_n + a \mu\; \partial_s u+ {v\over 4\ell}(\partial_q u_n)^2.$$
4. you can solve it a backward way since we have the terminal condition. It will give you $v_{n+1}$.

Now you can iterate. It is often talk the Howard method since it was explained in Howard, R.: Dynamic Programming and Markov Processes; MIT Press, Cambridge (1960).

What are the take-away:

• an HJB is a combination of a supremum (corresponding to solving the optimal control) and some PDE terms.
• once you solve the supremum, you get a PDE that is backward (that is natural: in control optimality comes from a backward reasoning).
• you can iterate and it will converge (moreover Howard tells you you converge a monotonic way: at each iteration you are closer to the true value function and the true optimal policy).

You can make the relation with Reinforcement Learning but it is not that true, since in RL the convergence is not monotonic. Nevertheless you can use RL to solve optimal trading problems, cf Section 5.4 of Mounjid, Othmane, and Charles‐Albert Lehalle. "Improving reinforcement learning algorithms: towards optimal learning rate policies." Mathematical Finance (2019).