0
$\begingroup$

I have been attempting to develop an algorithmic trading agent for a single asset pair and upon researching, it seems as if, in theory, reinforcement learning would be a natural way to approach this problem.

My idea was to have our observations be defined as follows:

$$o_i = (t_i, v_i, w_i, b_i)$$

where

  • $t_i$: time of observation

  • $v_i$: amount held by agent of asset 0

  • $w_i$: amount held by agent of asset 1

  • $b_i$: some actionable representation of the market such as quote data or order book data for an order book market, asset reserve levels for an automated market maker decentralized exchange marker, etc.

and to assume these observations are observations from a partially-observable markov decision process (described here), where the actions are defined as trading some of either asset 0 or 1 or doing nothing, so that the literature on those can be applied.

This would allow us to then implicitly model many real-world intricacies such as orders being executed before ours is executed given that we received a new observation by running our agent live and having it learn from the actual market by giving it a small amount of actual capital to work with so that it may explore and learn through actions based on the observations, perhaps pointing us more towards other insights about the market.

I am looking for what problems this approach may run into as I know I am very likely not the first to approach trading this way, e.g. this may sound good in theory, but the practical implementation is a lot more intricate, etc.

Improve this question
$\endgroup$
5
  • 1
    $\begingroup$ If you train your agent in real trading, the problem could be the algo will need a very long time to converge (assume it could), and it could cost you a lot of money. If you train it in backtesting/simulation, you need to have good/realistic order fill sims and market impact. Also, your action space seems a bit vague (you said "trading some"). You probably need clearer actions. And you did not mention what reward you want to train your agent on, which is just not a trivial thing to ignore. You should have good reward function and perhaps a decent amount of reward shaping. $\endgroup$
    – autoencoder
    Commented Jan 17 at 1:20
  • $\begingroup$ Thank you for the response @autoencoder those are good points, especially the convergence point since that wasn't something I've thought of too much. Also, good point on the action space, more specificity on my end would probably help. On the reward front, I think it would probably end up being some functional of the cumullative return over some period. $\endgroup$
    – QMath
    Commented Jan 19 at 20:36
  • $\begingroup$ @autoencoder Do you have an idea how one would address the potential impact of our order on future order book states if we're not doing live trading development? Since we can usually get a good approximation to order fills when using order book data, but it isn't clear to me how one would propagate the changes to future order book states based on our order when using historical data unless we had historical data that included us actually placing orders whereas with the live trading development framework, we would see the impact of our orders on future order book states. $\endgroup$
    – QMath
    Commented Jan 19 at 20:46
  • 1
    $\begingroup$ have a look at this question quant.stackexchange.com/questions/3551/… and my answer. $\endgroup$
    – lehalle
    Commented Jan 20 at 13:02
  • $\begingroup$ Great, thank you @lehalle ! I will have to piece through that. $\endgroup$
    – QMath
    Commented Jan 22 at 19:52

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.