# Optimal execution and reinforcement learning

Suppose a fairly simple problem: You have to buy (resp sell) a given number of shares V in a fixed time horizon H with the aim to minimize your capital spent (resp maximize your revenue).
There are some research papers on the web that claim that using reiforcement learning algorithms can help make decision. See for example:

1/ Nevmyvaka and Kearns: Reinforcement Learning for Optimized Trade Execution


These papers employ dynamic allocation strategies based either on limit order book or bid ask spreads to do so. Contrary to the classical paper

2/Almgren and Chriss: Optimal Execution of Portfolio Transactions


they do not assume a the security prices dynamic from which they derive their strategy. Instead they use backtest results on a test set to measure the performance of their program. Of course this also supposes constraining hypothsesis on the Limit Order Book dynamics which are difficult to test.

My question is twofold : Do you know good research papers using Reinforcement Learning (or other Machine Learning method) for this problem ? By good I mean that the test set is large (not just a few days of backtest) and there is a real effort to be clear about the hypothesis and to have as little as possible.

Has any of you applied it in a live trading environment or know someone who did?

• I like the question, but please avoid the acronyms like "ML", "RL", it makes it difficult to read. I corrected the onea I knew but "LOB"? no idea. You could also create links for the papers to improve the quality of the question. – SRKX May 29 '12 at 9:18
• @SRKX Sorry you are right. +1 – Zarbouzou May 29 '12 at 9:28

First, we are few quants and academics to use the full toolkit of machine learning: stochastic algorithms, to optimal trading. Here are at least two papers:

Kearns and his co-authors are also providing a lot of useful research.

Our approach is not only to try some machine learning techniques, but also to use the powerful mathematical tools that allowed to show their efficiency to prove that some algorithms are converging to optimal solutions.

More quantitatively, most of machine learning come from a on-line gradient descent on a given criteria, giving birth to a stochastic gradient descent. The stochasticity comes from this:

1. You want to minimize $\mathbb{E}||y-f_{\theta}(x)||^2$ with respect to $\theta$
2. If you build: $$\theta(n+1)=\theta(n)-\gamma(n)\times \frac{\partial \mathbb{E}||y-f_{\theta}(x)||^2}{\partial\theta(n)}$$
3. Then if it exists, $\theta(\infty)$ is a potential minimum for the criteria define at step 1
4. Now just build $\theta(n)$ simultaneously with the observation of pairs $(x_n,y_n)$: $$\theta(n+1)=\theta(n)-\gamma(n)\times \frac{\partial ||y_n-f_{\theta}(x_n)||^2}{\partial\theta(n)}$$
5. Under some ergodicity conditions, the limit of this $\theta$ will be the same than the previous (batch) one (you also need that $\sum_n \gamma(n)>\infty$ and $\sum_n \gamma(n)^2<\infty$; it is the celebrated Robbins-Monro theorem).

It is really suited for algo trading, but you need to apply this approach not blindly to any stochastic process $(x_n,y_n)$, but to ergodic ones.

Order flow relatively to the mid point seems to be more ergodic the the price itself, consequently it should be more efficient to use machine learning on intraday data rather than on daily ones.