So I've been studying the paper "Learning To Trade via Direct Reinforcement" Moody and Saffell (2001) which describes in detail how to use exponential moving estimates (EMAs) of returns at time t (r_t) to approximate both the Sharpe and Sortino ratios for a portfolio or security.

Note: in the paper he refers to the Sortino ratio as the "Downside Deviation Ratio" or DDR. I'm quite certain that mathematically speaking there is no difference between the DDR and the Sortino ratio.

So, the paper defines two values used to approximate either ratio, the Differential Sharpe Ratio (dsr) and the Differential Downside Deviation Ratio (d3r). These are calculations which both represent the influence of the trading return at time t (r_t) on the Sharpe and Sortino ratios. The EMAs used to calculate the DSR and D3R are based on an expansion around an adaption rate, η.

He then presents an equation by which I should be able to use the DSR or D3R at time t to recursively calculate a moving approximation of the current Sharpe or Sortino ratios at time t without having to perform a calculation over all t to get the exact result. This is very convenient in an environment with an infinite time horizon. Computationally, the data eventually would get too big to recalculate the full Sharpe or Sortino ratio at each timestep t if there are millions of timesteps.

$$S_t |_{\eta>0} \approx S_t|_{\eta=0} + \eta\frac{\partial S_t}{\partial \eta}|_{\eta=0} + O(\eta^2) = S_{t-1} + \eta\frac{\partial S_t}{\partial \eta}|_{\eta=0} + O(\eta^2)$$ $$D_t \equiv \frac{\partial S_t}{\partial \eta} = \frac{B_{t-1}\Delta A_t - \frac{1}{2}A_{t-1}\Delta B_t}{(B_{t-1} - A_{t-1}^2)^{3/2}}$$ $$A_t = A_{t-1} + \eta \Delta A_t = A_{t-1} + \eta (R_t - A_{t-1})$$ $$B_t = B_{t-1} + \eta \Delta B_t = B_{t-1} + \eta (R_t^2 - B_{t-1})$$

Above is the equation to use the DSR to calculate the Sharpe ratio at time t. To my mind, larger values of η might cause more fluctuation in the approximation as it would put more "weight" on the most recent values for r_t, but in general the Sharpe and Sortino ratios should still give logical results. What I instead find is that adjusting η wildly changes the approximation, giving totally illogical values for the Sharpe (or Sortino) Ratios.

Similarly, the following equations are for the D3R and approximating the DDR (a.k.a Sortino ratio) from it:

$$DDR_t \approx DDR_{t-1} + \eta \frac{\partial DDR_t}{\partial \eta}|_{\eta=0} + O(\eta^2)$$ $$D_t \equiv \frac{\partial DDR_t}{\partial \eta} = \\ \begin{cases} \frac{R_t - \frac{1}{2}A_{t-1}}{DD_{t-1}} & \text{if $R_t > 0$} \\ \frac{DD_{t-1}^2 \cdot (R_t - \frac{1}{2}A_{t-1}) - \frac{1}{2}A_{t-1}R_t^2}{DD_{t-1}^3} & \text{if $R_t \leq 0$} \end{cases}$$ $$A_t = A_{t-1} + \eta (R_t - A_{t-1})$$ $$DD_t^2 = DD_{t-1}^2 + \eta (\min\{R_t, 0\}^2 - DD_{t-1}^2)$$

I wonder if I'm misinterpreting these calculations? Here's my Python code for both risk approximations where η is self.ram_adaption:

def _tiny():
    return np.finfo('float64').eps

def calculate_d3r(rt, last_vt, last_ddt):
    x = (rt - 0.5*last_vt) / (last_ddt + _tiny())
    y = ((last_ddt**2)*(rt - 0.5*last_vt) - 0.5*last_vt*(rt**2)) / (last_ddt**3 + _tiny())
    return (x,y)

def calculate_dsr(rt, last_vt, last_wt):
    delta_vt = rt - last_vt
    delta_wt = rt**2 - last_wt
    return (last_wt * delta_vt - 0.5 * last_vt * delta_wt) / ((last_wt - last_vt**2)**(3/2) + _tiny())

rt = np.log(rt)

dsr = calculate_dsr(rt, self.last_vt, self.last_wt)
d3r_cond1, d3r_cond2 = calculate_d3r(rt, self.last_vt, self.last_ddt)
d3r = d3r_cond1 if (rt > 0) else d3r_cond2

self.last_vt += self.ram_adaption * (rt - self.last_vt)
self.last_wt += self.ram_adaption * (rt**2 - self.last_wt)

self.last_dt2 += self.ram_adaption * (np.minimum(rt, 0)**2 - self.last_dt2)
self.last_ddt = math.sqrt(self.last_dt2)

self.last_sr += self.ram_adaption * dsr
self.last_ddr += self.ram_adaption * d3r

Note that my rt has a value that oscillates around 1.0 where values >1 mean profits and <1 mean losses (while a perfect 1.0 means no change). I first make rt into logarithmic returns by taking the natural log. _tiny() is just a very small value (something like 2e-16) to avoid division by zero.

My problem(s) are:

  1. I would expect the approximated Sharpe and Sortino ratios to fall in the range 0.0 to 3.0 (give or take) and instead I get a monotonically decreasing Sortino ratio, and a Sharpe ratio that can explode to huge values (over 100) depending on my adaption rate η. The adaption rate η should affect noise in the approximation but not make it explode like that.
  2. The D3R is (on average) negative more than it is positive, and ends up approximating a sortino ratio that falls in a near-linear manner, which if left to iterate for long enough can reach totally nonsensical values like -1000.
  3. There are occasionally very large jumps in the approximation which I feel could only be explained by some error in my calculations. The approximated Sharpe and Sortino ratios should have a somewhat noisy but steady evolution without massive jumps such as those seen in my graphs.

Monotonically decreasing Sortino ratio due to on-average negative D3R Huge explosion of Sharpe ratio with relatively high adaption rate η=0.1

Finally, if someone knows where I could find other existing code implementations wherein the DSR or D3R is used to approximate the Sharpe/Sortino ratios it would be much appreciated. I was able to find this page from AchillesJJ but it doesn't really follow the equations put forth by Moody, as he is recalculating the full average for all previous timesteps to arrive at the DSR for each timestep t. The core idea is being able to avoid doing that by using the Exponential Moving Averages.

  • $\begingroup$ Note: my η is typically 0.007 $\endgroup$ Commented Aug 23, 2020 at 13:12
  • $\begingroup$ Also, if someone could confirm that I'm right about the DDR being exactly the same as the Sortino ratio, that would be lovely. As far as I can tell, they're the same. $\endgroup$ Commented Aug 23, 2020 at 13:30
  • $\begingroup$ Devil's advocate: why approximate Sharpe and Sortino? If you have a return TS you can calculate each directly $\endgroup$
    – Chris
    Commented Aug 23, 2020 at 19:00
  • $\begingroup$ @Chris there are a few answers to this, but mostly it's to do with computational efficiency: say I want to graph the evolution of the portfolio Sharpe ratio over time; as I keep aggregating time periods to the array I have to iterate over the entire array to recalculate the Sharpe ratio for each timestep. With millions of periods, this is infeasible. Another reason would be to use the DSR or D3R calculations to as part of a loss function in a neural network, in which case you need the risk-adjusted marginal utility only at time t. $\endgroup$ Commented Aug 23, 2020 at 19:13
  • 1
    $\begingroup$ Yes and no. The DSR statistic is inclusive of distant history in a similar way to EMA being inclusive of distant returns...albeit with very low weightings; standard Sharpe calcs obviously don't do that. I've only had a cursory look but they don't really approximate Sharpe/Sortino so much as they're related calculations that the authors chose to use in their approach. Nothing wrong with that (and I don't have anything more specific to add to your original question) really. $\endgroup$
    – Chris
    Commented Aug 23, 2020 at 20:13

3 Answers 3


For anybody still following this:

I figured out that the equations and my code work fine; the problem was that I had to scale the returns before doing the risk calculations to avoid float32 precision data loss, and also just that my value for η was far too high. Lowering my η value to <= 0.0001 produces totally logical sharpe and sortino approximations. As a sidenote, this also allows my neural network to learn directly from the marginal sharpe and sortino calculations, which is great.

As well, using logarithmic returns was problematic for the sortino approximation, so I effectively changed it to rt = (rt - 1) * scaling_factor which makes the sortino approximation not tend towards negative values anymore.

Logarithmic returns would have worked fine if my only goal was to use the DSR/D3R as a loss calculation in my neural network, but to get good sortino approximations it doesn't work as it sharply emphasizes negative returns and smooths positive returns.


If your concern is about computational efficiency in calculating Sharpe/Sortino over large and increasing amounts of data, you can use incremental/online methods to calculate means, standard deviations etc. over the whole data set. Then just use the latest, online calculated value for the Sharpe/Sortino of the whole data set. This will avoid the problem of older data having less weight than newer data, which is implicit when using EMAs.

My answer on the Data Science SE at https://datascience.stackexchange.com/questions/77470/how-to-perform-a-running-moving-standardization-for-feature-scaling-of-a-growi/77476#77476 gives more detail and a link.

  • $\begingroup$ While this does help, computational efficiency is unfortunately only a part of the concern; the other half is that ideally I'd like to use the DSR or D3R as components in a loss calculation for a neural network, where it's essential to calculate the risk-adjusted marginal utility only at time t. It would be computationally elegant to be able to "kill two birds with one stone" and approximate Sharpe/Sortino based on those incremental values. $\endgroup$ Commented Aug 24, 2020 at 8:17
  • $\begingroup$ I read the paper you posted; I'm going to try and implement this to see if I can get similar results from it, and if so, will mark as correct answer. $\endgroup$ Commented Aug 24, 2020 at 16:38

This has been really, really useful, thank you. I've applied this to an RL algorithm (just the DSR metric) and I have a few things to ask if this thread is still active.

  1. What do you do about the first steps? it seems like the values are unstable at the beginning of the sequence.

  2. Also, at what values would you initiate the moving averages?

  3. I've also experienced sudden drop during training enter image description here

Why do you think that is?

Here's your code, just changed the naming and put it into a class, I hope I did it right

class DifferentialSharpeRatio: def init(self, eta=1e-4): self.eta = eta self.last_A = 0 self.last_B = 0

def _differential_sharpe_ratio(self, rt, eps=np.finfo('float64').eps):
    delta_A = rt - self.last_A
    delta_B = rt**2 - self.last_B

    top = self.last_B * delta_A - 0.5 * self.last_A * delta_B
    bottom = (self.last_B - self.last_A**2)**(3 / 2) + eps

    return (top / bottom)[0]

def get_reward(self, portfolio):
    net_worths = [nw['net_worth'] for nw in portfolio.performance.values()][-2:]
    rt = pd.Series(net_worths).pct_change().dropna().add(1).apply(np.log).values

    dsr = self._differential_sharpe_ratio(rt)

    self.last_A += self.eta * (rt - self.last_A)
    self.last_B += self.eta * (rt**2 - self.last_B)

    return dsr
  • $\begingroup$ You should post this as a new question and provide a link to this thread, rather than posting as an "answer" with additional questions. $\endgroup$ Commented Jan 29, 2021 at 10:15
  • $\begingroup$ I understand. Apologies. $\endgroup$
    – orie
    Commented Jan 30, 2021 at 19:27
  • $\begingroup$ @Attack68 might help. $\endgroup$ Commented Jan 30, 2021 at 19:48
  • $\begingroup$ Or @BobJansen... (in moving the answer to a new question) $\endgroup$ Commented Jan 30, 2021 at 19:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.