I want to know what are some commons methods/tricks people use to analyze the quality of their trading signals.

Let's say I have a dataframe with all my different signals and their performance. These signals have been executed in different market conditions, on different instrument...

So to understand better on which segment the signals are performing properly and on which segment the signals are performing badly it gets complicated:

  • we can first look at the performance on each instrument
  • then on each sector
  • then on different quantile of market volatility
  • then on ...

and then you want to understand what's the performance on combination of the split (on that sector when the volatility is bigger than...).

So more generally when you want to analyze your trading signals based on a number of $n$ factors how do you go about getting a better understanding of: in which conditions the signal perform well and in which conditions does it perform badly?

Because the problem I am facing is that the number of segments to analyze can be extremely large so I get lost very easily in the analysis, and also when can you consider that would you observed is statistically significant and is not just "chance".

  • 1
    $\begingroup$ all I can say is that, if you break things down too finely, you'll find some kind of pattern ( say some kind of pattern in sector performance ) and that will cause you to possibly use sector-specific parameters since the sector performance varies. But I would be careful about doing things like that because it can lead to data mining-overfitting. Sometimes, the most general parameter is the most stable. $\endgroup$
    – mark leeds
    Mar 10 at 6:56

2 Answers 2


It sounds like you are trying to derive a relationship between performance and underlying market dynamics so that you know when and where to deploy a particular signal or set of signals. The nature of the problem boils down to this. How does one extract a signal from the market that tells me what signals will work at a particular moment in time. And the answer is that you might be able to arrive at a set of criteria that works today but will not work tomorrow. The amount of time I have spent working on this very problem makes me a bit sad because, while one can learn a lot going down this path, one cannot find the answer in this form.

The other observation from your question is that you are probably on the path of over fitting. When you start to get so fine grained in your analysis this is often the result. Keep things simple. Look at one market and find something that works for that market. Then scale to other markets. Keep your analysis strictly on no more than second-moment level. Going beyond that is overfitting.

You mention that you have your signals and performance in a Dataframe so you are likely using python. I find that Jupyter Notebook is a fantastic way to do simple what-if analysis and matplotlib has tools that let you easily visualize placement of 100s of trade entry and exit alongside your signals and market data to see exactly what works and what doesn't under various market conditions (trending, consolidating, etc.).

The last item worth mentioning because it is often overlooked is data integrity. Where are you getting your market data? The best place to get high fidelity, accurate data is directly from the exchange. The worst place is free data scraped from the web. Not much better is data from retail providers such as IBKR. You cannot build a successful trading algorithm from bad data.


To analyze trading signals based on a set of n factors and determine under which conditions the signals perform well or poorly, I use a very simple two-step approach:

1.Choose the type of loss function depending on whether you are focusing on statistical losses (such as mean absolute error) or economic losses (such as those proposed by Granger, 1999, including forecast trading return and correct forecast direction). The choice is wide and allows flexibility in later analysis, but is tied to your exact application or intention of how to trade the signal.

2a. Run a linear regression or decision tree using your loss data as the dependent variable and the n factors as the explanatory variables. This analysis will help identify factors that explain or predict declines in the predictive quality of your trading signals.

2b. In cases where there are multiple competing models, compare them to determine under what conditions (e.g., higher VIX?) one outperforms the other. The dependent variable here is the difference (usually absolute or squared) between the losses of the two models. The method then follows the same approach as in 2a.

These methods are inspired and adapted from Giacomini & Rossi's (2009) Forecast Breakdown Test and Giacomini & White's (2006) Conditional Predictive Ability Test. You can effectively use the Forecast Breakdown Test to predict or explain forecast breakdowns in your models, which I like a lot for "in production" algorithms.


  • Giacomini, R. and Rossi, B., 2009. Detecting and predicting forecast breakdowns. The Review of Economic Studies, 76(2), pp.669-705.
  • Giacomini, R., White, H., 2006. Tests of conditional predictive ability. Econometrica 74, 1545–1578.
  • Granger, C. W. J., Empirical Modeling in Economics: Specification and Evaluation (London: Cambridge University Press, 1999).

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.