3
$\begingroup$

One of the main advantages of (deep) reinforcement learning approaches (compared to more widely known supervised deep learning approaches) is the fact that it enables us to automatically take sequentiality into account. It's clear that the optimal action at time $t$ doesn't necessarily have to be the one that maximizes the expectation of immediate reward (greedy is not necessarily optimal in the long run). Therefore the framework of (D)RL seems appropriate for portfolio optimization where we are interested in maximizing a certain objective (say Sharpe's ratio) over a longer period.

However, many papers that deal with (D)RL applications in portfolio optimization use historical market data to build a deterministic MDP to train the model on. Under such approaches the state at time $t$ is a often list of historical returns for a chosen set of assets. Such an MDP deterministic in the sense that
1) the state at $t+1$ ($s_{t+1}$) will not depend on action at $t$ ($a_t$) since it consists of (already fixed) historical data
2) the reward at $t+1$ ($r_{t+1}$) will be a deterministic function of the previous action and state ($a_t, s_t$)

Therefore, the optimal action at $t$ will be greedy (since whatever we do it will not affect the next state) and the advantages of the DRL approach seem to be gone while its disadvantages (sample inefficiency, instability, etc.) are still there.

My question is the following: Why should one even try to use (deep) reinforcement learning for portfolio optimization when given historical market data (i.e. deterministic MDP to train on)?

Addendum: It's clear to me that we can artificially introduce sequentiality by say including current portfolio weights in the state vector (for the sake of say taking into account trading costs), but it still seems to me that a) under small trading costs the optimal action will be close to greedy, thereby still not achieving full sequentiality as desirable for RL based approaches b) many researchers who use DRL approaches totally ignore trading costs and don't seem to be bothered by what I've outlined above

$\endgroup$
2
  • $\begingroup$ what do you mean by reinforcement learning "automatically taking sequentiality into account"? and what does deterministic MDP stand for $\endgroup$
    – develarist
    Nov 22, 2019 at 21:44
  • $\begingroup$ MDP = Markov Decision Process $\endgroup$
    – jacques
    Jun 27, 2020 at 11:44

1 Answer 1

1
$\begingroup$

Whether you will end up with a deterministic or stochastic MDP fully depends on what you treat as the state and/or reward. Even working with fixed historical price data, different actions (sequence) will lead to totally different terminal wealth/Sharpe ratio, making it legitimate, or even favorable, to apply RL. You can even adopt bootstrapping type of training with fixed real data to enhance the randomness in the RL problem. Two good reference papers can be found here:

https://arxiv.org/abs/1904.11392

https://arxiv.org/abs/1907.11718

Hope this helps!

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.