I am testing various technical trading rules (TTR) on the cryptocurrency market.

I have already setup some significance tests, to compare the returns and volatilities.

I would now like to test it from a different angle using the ROC model, often applied when backtesting PD [Probability of Default] models. I am just not sure if this is actually feasible in the environment I am trying to apply it.

The logic I have applied until now is that if the output of my TTR is a "BUY" and the underlyings returns that day is bigger than zero, it is a TRUE BUY on the other hand if the return of the underlying is smaller than zero that day I get a FALSE BUY. Having this information I think I should be able to set-up a ROC model.

My question here would be, is this approach actually feasible? Has someone already taken this approach?

  • $\begingroup$ If the outcomes of the TTR are highly asymmetric (for ex. many small wins, a few big losses) then the high percentage of true buys may give a misleading indication of success, compared to the dollar profit. (Imagine for ex. a TTR of the form: buy at random times, with a 100 USD profit target and a 1000 USD stop loss...). $\endgroup$
    – Alex C
    Jul 28, 2018 at 18:23

1 Answer 1


So ROC performance metrics are common place in machine learning applications, particularly for classification tasks.

Looking at the way you have setup the question here you wont be able to plot a ROC curve as it relies on a continues output (not discrete). You would however be able to plot a confusion matrix and look at performance metrics such as precision, recall, and F1-score.

All you would need is the actual forecast vs your predicted forecasts.

I did this in python using Sklearn on a simple trend following strategy:

enter image description here

Note: This strategy I built here uses a secondary model to determine when to place a trade based on the primary technical model and that is why all of predictions are labeled as 1. It does however still illustrate the point.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.