After reading papers on reinforcement learning with respect to the problem of optimal execution (Nevmyvaka et al (2006), Ning et al (2018), etc), I was wondering if the Markov property assumed in all these papers is rational? Has there been at least an empirical comparison/paper detailing tradeoffs & performance of non-Markovian models with less state variables vs a Markov model with more state variables someone could direct me towards?

  • 1
    $\begingroup$ Do you have something in particular that doesn't make sense to treat as markovian? In general, for modeling one has to make simplifications in order to derive a solution, and the Markov property makes the problem a LOT easier. Sometimes it makes sense, sometimes it doesn't depending on the problem at hand. $\endgroup$ Dec 19, 2020 at 13:37
  • $\begingroup$ There's a sizeable class of literature/research on non-Markovian reward systems outside this problem, so I guess I'm just looking for comparison of models between for example trying to make a system Markov by adding more state variables like bid-ask spread, signed volume, etc. vs non-Markovian system w/ less state variables $\endgroup$ Dec 19, 2020 at 15:23


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.