In Advances in Financial Machine Learning, Lopez explains how we should build a primary exogenous model (binary classifier) to identify trading opportunities and a secondary meta model to filter out the false positives from the exogenous model. The exogenous model should have high recall to identify most trading opportunities at the expense of a low precision (high number of false positives). The idea of the meta model is to "increase your F1-score by filtering out the false positives, where the majority of positives have already been identified by the primary model."

It is clear we should train the primary model such that it maximises recall and / or choose a probability threshold that yields high recall at the expense of low precision. However, Lopez does not explain which metric we should maximise / minimise when training the secondary model to "filter out the false positives".

I assume a scheme outlined by Hudson and Thames:

  1. Train a primary model on the original data features
  2. Choose a probability threshold which yields high recall
  3. Use the primary model to make predictions on the training data with the threshold from step 2.
  4. Append the predictions from step 3. to the original data features. This is a new feature for the secondary model
  5. Define the meta labels for the secondary model as follows: If the primary model’s predictions matches the actual values, then we label it as 1, else 0. This part is important in how we define labels for the secondary model
  6. Train the secondary model using the features and labels constructed from the steps above

I have 2 questions

i) Is the above method in step 5. of generating meta labels correct? Lopez did not make it explicit, but in the book he does say "... we correct for the low precision by applying meta-labeling to the positives predicted by the primary model" which seems to indicate step 5. above is correct

ii) What metric should we maximise / minimise when training the secondary model so that indeed false positives are filtered out? Again, Lopez is not explicit here; he only says that the primary model should have high recall.

  • $\begingroup$ I would email him. I've emailed him a few times and he's quite responsive. His email is probably on the internet. $\endgroup$
    – mark leeds
    Mar 8 '20 at 21:51
  • $\begingroup$ @markleeds good idea - I've done that, let's see what he says. Maybe I can answer the question myself if / when he replies $\endgroup$
    – PyRsquared
    Mar 9 '20 at 16:15
  • $\begingroup$ I'm pretty confident that he'll reply. I have the book but haven't read it. Do you like it ? Good luck. $\endgroup$
    – mark leeds
    Mar 10 '20 at 2:04
  • $\begingroup$ @markleeds Indeed he did reply, I'm surprised he was so fast. But the answer he gave was quite short - he basically said optimize F1 for the meta model, and the meta labels should be generated as I have in the question. I have the book and think it's excellent for the most part - it just needs to be expanded / more explicit in parts like this. $\endgroup$
    – PyRsquared
    Mar 10 '20 at 8:41
  • $\begingroup$ Sometimes ( don't know if that's the case here ), a short answer has to do with the proprietary nature of what they do in that unclear step. People who write books obviously want to be as opened as possible without providing the information that might make a difference in success or failure. So, don't ever purchase a book if you're expecting it to provide the secret sauce. It never will. The author will generally not come out and say this explicitly. Again, I can't speak for your particular case but what I've said is often true.. $\endgroup$
    – mark leeds
    Mar 10 '20 at 19:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.