2
$\begingroup$

The chart below is the 15-minute EURUSD from earlier today. The blue lines represent dividers between three subjective but reasonable segments that can easily be made out by the eye. I would characterize the three segments as:

  1. Low volatility (mostly) neutral/horizontal trend
  2. Higher volatility neutral or slight down trend
  3. Higher volatility linear up trend

enter image description here

I'm wondering if there are any existing algorithms or methods that could be used to identify dividing points between segments. By comparison, it's relatively straightforward to divide a chart into up and down swings given some minimum price increment but dividing based on multiple factors (trend and volatility in this case) seems more difficult. In theory, one could use any number of characteristics, including indicators. Something like comparing multiple moving averages would be easy, whereas determining where real-time volatility changes is not as easy (e.g., one large bar after a series of small might just be one outlier rather than a change in market dynamics).

I'm not necessarily concerned about being able to do this perfectly in real time. Again for something like volatility, it would likely take multiple bars to know that the dynamic has changed, but even being able to do this on historical data might be helpful in terms of backtesting different strategies.

Bounty Update: I've read about a number of different algorithms but can't seem to find one that will best capture what I'm looking for here. I think I could use a dynamic sliding window algorithm that uses spikes in Wasserstein distance that exceed a threshold, but I haven't seen anything that says that specifically. I'd really like to be able to do this over a year of minute data so calculation time is a factor.

There's so much scattered information about time series segmentation so I'm hoping someone can give some more direct advice.

Improve this question
$\endgroup$
6
  • $\begingroup$ I think you would need to start by thinking about specifc thresholds for what you consider as volatile and part of a trend for example. That way you lay the ground work for a potential ML classification task. $\endgroup$
    – Hamish Gibson
    May 8, 2020 at 15:30
  • 2
    $\begingroup$ Hint: start playing with simple partitional clustering techniques, like K-means, applied to price according to some features among which there must be a time index. $\endgroup$
    – Lisa Ann
    May 9, 2020 at 8:18
  • $\begingroup$ I’ve come across some K-means techniques but trying to get a grasp of which options are best. I didn’t realize how much information was out there about time series segmentation. Regarding the time index, wouldn’t this be very sensitive to my selection of K? If I understand your suggestion correctly, if K is too large, the algorithm might try to divide sections that are homogeneous to the naked eye or fail to divide heterogeneous sections if K is too small. The other issue is that a lot of these algorithms seem to be O(n^2) or worse, which could be problematic for minute data. $\endgroup$
    – SuperCodeBrah
    May 9, 2020 at 8:34
  • $\begingroup$ Yes, you need some tuning process of K-means hyperparameters in order to solve the trade-off. But then, you'll probably switch to better clustering methods, like DBSCAN. $\endgroup$
    – Lisa Ann
    May 13, 2020 at 9:00
  • $\begingroup$ is there a source from where you got the idea of "dynamic sliding window algorithm that uses spikes in Wasserstein distance that exceed a threshold"? $\endgroup$
    – develarist
    Nov 4, 2020 at 17:13

4 Answers 4

Reset to default
1
$\begingroup$

I don't know, because your chart shows no time stamps, but I'd hazard a guess that the first third is the "overnight" session after the New York close until the London open, the second third is the London open until the London close, and the last third is the remaining part of the trading day until the New York close. If this is so, you don't necessarily need an algo to identify high/low volatility or trends, but can make use of some stylised "facts." A few might be:-

  1. The "overnight" session will always tend to exhibit low volatility compared to London/New York trading hours.

  2. Most of the time there will be a change of market type/direction on both the London and New York opens as new participants begin trading, with increases in volume at both times, compared to what has been the norm for the previous few hours beforehand.

  3. Ranges/support/resistance/trends set up in the immediately preceding session hours can provide hints towards future market types/direction on breaks and bounces etc.

My point being it's not necessary to throw computing power at this problem, but rather it is a problem of market understanding, based on some research of course.

Improve this answer
$\endgroup$
1
$\begingroup$

You could look into the trend-scanning method which is described in a new book by Marcos Lopez de Prado, "Machine Learning for Asset Managers". Essentially you fit a linear regression to multiple forward looking periods of increasing length (say you scan from 5 periods ahead, to 50 periods ahead) and choose the regression fit with the highest slope adjusted for standard error of the fit (Slope / SE) or t-statistic. The intent is to let trends run to the extent that they are more 'statistically significant' than others in the test, meaning they have less potential to be due to random chance or noise.

This isn't exactly what you want but it is along the same lines, I think. The issue is that lower volatility would typically result in a higher t-stat given an equivalent slope (trend) and you actually want to capture changes in volatility as well.

Regarding clustering you could cluster summary statistics from the distribution of returns in a 2-dimensional instead of a 1-dimensional space. Use volatility (or variance) as the 1st dimension and skewness as the 2nd dimension. Thus you are clustering 2D points in a 2D space along both axes, and including positive or negative skew (more positive versus negative returns or vice versa).

This could capture at least 4 overall 'categories' or dimensions of the return distribution and at different granularities depending on how hyper-parameters are tuned,

  1. Low volatility, negatively skewed
  2. Low volatility, positively skewed
  3. High volatility, negatively skewed
  4. High volatility, positively skewed
Improve this answer
$\endgroup$
0
$\begingroup$

Why not using a volatility indicator if volatility is the only factor? set 2 thresholds and you are done.

Separately recommend the read of a time series analysis or econometrics book, which will provide you with a deeper understanding.

Improve this answer
$\endgroup$
1
  • $\begingroup$ If I used a volatility indicator, I think I'd have to do some sort of averaging/smoothing, which means that rather than a hard cutoff, the indicator would slowly start to increase in times of increased volatility. I suppose I could find highs/lows in the indicator (similar to how I could segment by finding highs and lows in price), but this still presents an opportunity for inaccuracy due to the averaging effect. I'm not sure it's possible to achieve what I'm looking for without long calculation times to determine the segments. $\endgroup$
    – SuperCodeBrah
    May 15, 2020 at 19:04
0
$\begingroup$

Seems you're trying to segment based on volatility and trend which might be two different/separate problems. Googling 'change point detection' might be a good start (or see https://arxiv.org/abs/2003.06222). There are tests which test for both changes in location and scale (i.e. trend and vol).

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.