I'm attempting to optimize a reinforcement learning system to maximize risk adjusted returns. I have currently defined the reward as the differential Sharpe ratio at each step: the influence of the return at time t on the cumulative Sharpe ratio.
This was defined in this paper [Reinforcement Learning for Trading, by John Moody and Matthew Saffell, NIPS, 1999], however I am interested in maximizing Sortino ratio instead. The assets I am trading are highly volatile but also heavily skewed towards positive returns; the Sharpe ratio of the strategy is only 2.5 but the Sortino is ~11. So, maximizing for Sharpe here doesn't make much sense. The derivation of this differential Sharpe ratio is:
I would like to adapt this only take into account the variance of my downside risk. Any help or further reading would be great.