1
$\begingroup$

I'm working on a balanced, binary classification problem in a time-series (financial) dataset. I am using K-fold cross validation that is adapted for time-series (so that I'm never using future data to predict past data).

I have tried many algorithms, such as SVM, RandomForest and K-Nearest Neighbors. While all of them can achieve good results in cross validation, NONE of them have generalized well to the test set.

I use the cross validation to run grid-search feature selection and hyperparameter tuning simultaneously to find the best combination, but again - I have not achieved any generalization.

Do you have any ideas as to why this might be? Any general advice for dealing with this kind of scenario?

Improve this question
$\endgroup$
1
  • 2
    $\begingroup$ Financial time-series are notoriously noisy and have very little information content in them for the purpose of machine learning methods. At least, that is my opinion. $\endgroup$
    – rubikscube09
    Mar 2, 2021 at 14:37

1 Answer 1

Reset to default
2
$\begingroup$

One obvious answer, but not the one you'll probably want to hear is that maybe there is no single distribution for the variable you're trying to explain. It might be for want of a less-bad metaphor "regime-driven".

Being simplistic for simplicity's sake, imagine you were looking at the relationships between stock, bond, and commodity markets. The period between ~2002-~2007/8 might be described as "China-on, China-off"; that 07/08 and 12/14 as "risk-on, risk-off"; 12/14 and 18/20 as "liquidity-on, liquidity-off"; and who knows to nature of this now? :-)

It would then be very possible to train any model for any historical sample, and achieve attractively comfortable cross-validation results in your training set. However, generalisation in your test set might nevertheless stink, if the test set represented a different regime, with a different set of prevailing norms, assumptions, and/or trading rules.

I've certainly seen this in many of my own models over the last decade.

Improve this answer
$\endgroup$
8
  • $\begingroup$ Thank you for your answer! I understand what you mean, and I've done a little work to investigate the possibility of dataset shift. The reason I don't think this is the issue is because I've tried applying machine learning models to a variety of timescales (a year of daily data, a month of hourly data and a different month of hourly data) and I run into the issue every time - good cross validation results, poor test-set results. I think it's unlikely that the regime has shifted in all three of those examples, if that makes sense. What do you think? $\endgroup$
    – Vladimir Belik
    Jan 31, 2021 at 0:09
  • $\begingroup$ Hmm, well then you have my heartfelt apologies for being patronising. I hope you'll accept in mitigation that oh-so-many-people don't bother to cover these simple angles; and the "turn it off and on again" simplistic stuff often really is the right answer... Will ponder... $\endgroup$
    – demully
    Jan 31, 2021 at 0:47
  • $\begingroup$ You've used a variety of timescales of the same data, but have you tried adding more data - that'd be my next answer to prevent overfitting. $\endgroup$
    – pSrIoGcNeAsLs
    Jan 31, 2021 at 0:55
  • $\begingroup$ Yes, another way to approach this is to - temporarily - suspend the distinction between test and training. Simply for the purposes of identifying the discontuinity in the data (ie not for prediction purposes). So pick a date or a random series of dates, take the set of data a year before vs a year after. Run a model for both, and test on the other. And run a model on both combined. If generalisation is weak A vs B, B vs A, and your model for A&B is significantly weaker than for A and for B, then you are probably looking at a simple overfitting problem. Start trimming, until the gap closes... $\endgroup$
    – demully
    Jan 31, 2021 at 1:10
  • 1
    $\begingroup$ More formally, take a sample of two consecutive periods (A&B). Run your model in-sample, and take the predictions. Then run in-sample A|B and B|A. Take the change in predictions given the partition. Then measure the residuals of these to your target. Do an F-test of your partition. Is it significant? If it isn't, you're overfitting somehow. If it is, then you really shouldn't have a generalisation problem. ;-) $\endgroup$
    – demully
    Jan 31, 2021 at 2:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.