0
$\begingroup$

One of the main advantages of (deep) reinforcement learning approaches (compared to more widely known supervised deep learning approaches) is the fact that it enables us to automatically take sequentiality into account. It's clear that the optimal action at time $t$ doesn't necessarily have to be the one that maximizes the expectation of immediate reward (greedy is not necessarily optimal in the long run). Therefore the framework of (D)RL seems appropriate for portfolio optimization where we are interested in maximizing a certain objective (say Sharpe's ratio) over a longer period.

However, many papers that deal with (D)RL applications in portfolio optimization use historical market data to build a deterministic MDP to train the model on. Under such approaches the state at time $t$ is a often list of historical returns for a chosen set of assets. Such an MDP deterministic in the sense that
1) the state at $t+1$ ($s_{t+1}$) will not depend on action at $t$ ($a_t$) since it consists of (already fixed) historical data
2) the reward at $t+1$ ($r_{t+1}$) will be a deterministic function of the previous action and state ($a_t, s_t$)

Therefore, the optimal action at $t$ will be greedy (since whatever we do it will not affect the next state) and the advantages of the DRL approach seem to be gone while its disadvantages (sample inefficiency, instability, etc.) are still there.

My question is the following: Why should one even try to use (deep) reinforcement learning for portfolio optimization when given historical market data (i.e. deterministic MDP to train on)?

Addendum: It's clear to me that we can artificially introduce sequentiality by say including current portfolio weights in the state vector (for the sake of say taking into account trading costs), but it still seems to me that a) under small trading costs the optimal action will be close to greedy, thereby still not achieving full sequentiality as desirable for RL based approaches b) many researchers who use DRL approaches totally ignore trading costs and don't seem to be bothered by what I've outlined above

$\endgroup$
  • $\begingroup$ what do you mean by reinforcement learning "automatically taking sequentiality into account"? and what does deterministic MDP stand for $\endgroup$ – develarist Nov 22 at 21:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.