I came across a few instances of papers claiming to formulate a markov decision process for trading strategies trained on orderbook data. They claim solve it using reinforcement learning, but yet somewhere in the exposition they would make the assumption that their actions have zero impact on the system.

So my question is, what’s there to reinforcement learn, when your actions don’t impact the system?

Wouldn’t it end up to be merely “learn from history”? Meaning it would end up being trained such that it just picks what historically gives the best outcome?


Reinforcement learning, broadly speaking, doesn't require that your actions have an impact on the system. It only requires that there be states, actions that can be taken in those states (with potentially stochastic results), and rewards (again, potentially stochastic) that are the result of actions taken in particular states.

Also yes, it would learn from history. Is there something particularly wrong about that though?

  • $\begingroup$ Well, what about multi-agent RL? $\endgroup$ Jul 19 '18 at 3:39
  • $\begingroup$ I'd still maintain my position. $\endgroup$
    – Kevin Li
    Jul 19 '18 at 10:33
  • $\begingroup$ You're absolutely right in that it does not violate the problem setup for reinforcement learning. My main gripe with it is that of clarity. If actions don't impact the environment, then all I need to do is observe all the state transition probabilities, and that's enough for me to make decisions. One does not have to go through all the reinforcement learning stuff and talk about Bellman equation and all that... It's abit like saying oh I'm going to apply the complicated root finding algorithm that handles degree-6 polynomials, when in actually fact all my "polynomials" are degree-1 i.e. "ax+b" $\endgroup$
    – Danny
    Jul 20 '18 at 8:38
  • $\begingroup$ I suppose it depends on what you mean by "impact the environment"; it feels like you're taking a particular philosophical view on this. Assuming that you model your reinforcement learning problem as an MDP, the only thing that you care about is state transitions. Actions that "impact the environment" just change the state. $\endgroup$
    – Kevin Li
    Jul 20 '18 at 14:08
  • $\begingroup$ One needs to separate a number of things a) actions have impact on rewards b) actions have impact on observations c) reward model is unknown d) transition model is partially known etc ... generally people mean model free RL and actions have impact on rewards AND on actions. In trading, generally you know the reward model, you assume the trade impact is small and in the absence of market impact you observe everything (you get to observe all prices regardless of action). The impact of action only really affects the order book or the market if you are HUGE. $\endgroup$
    – mathtick
    Apr 1 '20 at 14:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.