Optimal execution and reinforcement learning

Question

Suppose a fairly simple problem: You have to buy (resp sell) a given number of shares V in a fixed time horizon H with the aim to minimize your capital spent (resp maximize your revenue).
There are some research papers on the web that claim that using reiforcement learning algorithms can help make decision. See for example:

1/ Nevmyvaka and Kearns: Reinforcement Learning for Optimized Trade Execution 

These papers employ dynamic allocation strategies based either on limit order book or bid ask spreads to do so. Contrary to the classical paper

2/Almgren and Chriss: Optimal Execution of Portfolio Transactions

they do not assume a the security prices dynamic from which they derive their strategy. Instead they use backtest results on a test set to measure the performance of their program. Of course this also supposes constraining hypothsesis on the Limit Order Book dynamics which are difficult to test.

My question is twofold : Do you know good research papers using Reinforcement Learning (or other Machine Learning method) for this problem ? By good I mean that the test set is large (not just a few days of backtest) and there is a real effort to be clear about the hypothesis and to have as little as possible.

Has any of you applied it in a live trading environment or know someone who did?

Answer 1

First, we are few quants and academics to use the full toolkit of machine learning: stochastic algorithms, to optimal trading. Here are at least two papers:

• Optimal split of orders across liquidity pools: a stochastic algorithm approach, Sophie Laruelle (PMA), Charles-Albert Lehalle, Gilles Pagès (PMA)
• Optimal posting distance of limit orders: a stochastic algorithm approach, Sophie Laruelle (LPMA), Charles-Albert Lehalle, Gilles Pagès (LPMA)

Kearns and his co-authors are also providing a lot of useful research (see Reinforcement learning for optimized trade execution).

Our approach is not only to try some machine learning techniques, but also to use the powerful mathematical tools that allowed to show their efficiency to prove that some algorithms are converging to optimal solutions.

More quantitatively, most of machine learning come from a on-line gradient descent on a given criteria, giving birth to a stochastic gradient descent. The stochasticity comes from this:

1. You want to minimize $\mathbb{E}||y-f_{\theta}(x)||^2$ with respect to $\theta$
2. If you build: $$\theta(n+1)=\theta(n)-\gamma(n)\times \frac{\partial \mathbb{E}||y-f_{\theta}(x)||^2}{\partial\theta(n)}$$
3. Then if it exists, $\theta(\infty)$ is a potential minimum for the criteria define at step 1
4. Now just build $\theta(n)$ simultaneously with the observation of pairs $(x_n,y_n)$: $$\theta(n+1)=\theta(n)-\gamma(n)\times \frac{\partial ||y_n-f_{\theta}(x_n)||^2}{\partial\theta(n)}$$
5. Under some ergodicity conditions, the limit of this $\theta$ will be the same than the previous (batch) one (you also need that $\sum_n \gamma(n)>\infty$ and $\sum_n \gamma(n)^2<\infty$; it is the celebrated Robbins-Monro theorem).

It is really suited for algo trading, but you need to apply this approach not blindly to any stochastic process $(x_n,y_n)$, but to ergodic ones.

Order flow relatively to the mid point seems to be more ergodic the the price itself, consequently it should be more efficient to use machine learning on intraday data rather than on daily ones.

[UPDATE in Jan 2020] I wanted to make an addition around two papers

• in Improving reinforcement learning algorithms: towards optimal learning rate policies, by Mounjid and L, we show the role of the learning rate in reinforcement learning and provide two illustrations in optimal trading: (1) optimal trade scheduling and (2) optimal placement in an orderbook. It shows that if you build carefully your algorithm, it can find the optimum. Moreover, we make the difference between 'original' RL (when you really discover the dynamics you have to control while you attempt to control them), and something that is closer to the old 'policy-value iterations' of Howard (see for instance Dynamic programming, Markov chains, and the method of successive approximations, by DJ White in 1964!) when you have access to a simulator. In the latter case, the exploration-exploitation aspect of the learning is not a big issue.
• in Learning a functional control for high-frequency finance by Leal, Laurière and L. We provide another way to extend optimal control of trading to machine learning: 'simply' write the optimal scheduling problem (for instance over 70 intervals of 5 minutes) with a closed loop controller emulated by a neural network, and wait for the (simulated) end of the day to back propagate over the 70 uses of your neural controller. No need to go through an approximation of a Q-function. It is straightforward and can be easily compared to a non-learning approach.