Determine trends of data (direction detection or turning point detection)

Question

I'm working on a model to determine trends (direction detection or turning point detection). Suppose that we have a stock trend which is illustrated below. Blue line is real trend of stock close prices and red line is imaginary approximation of the trend line.

I implemented different models such as Piecewise Linear Representation (PLR) or Wavelet to find the red line but I haven't got precise results so far. What is your recommendation for this problem?

Answer 1

The graph you attached suggests that you were trying to find swings between major highs and lows. This can be done by simply finding local extrema in the price series. The concept is:

1. find local extrema: minima in Low prices, maxima in High prices;
2. find local extrema in the results, if swings are too short;
3. repeat #2 until satisfied with the results.

This is the Python code to do that using Pandas data frame and Numpy's argrel* functions to locate relative extrema in series:

import pandas as pd
import numpy as np
from scipy.signal import argrelmin, argrelmax

N = 3 # number of iterations
df = pd.read_csv('prices.csv') # load your stock prices in OHLC format
h = df['High'].dropna().copy() # make a series of Highs
l = df['Low'].dropna().copy()  # make a series of Lows
for i in range(N):
        h = h.iloc[argrelmax(h.values)[0]] # locate maxima in Highs
        l = l.iloc[argrelmin(l.values)[0]] # locate minima in Lows
        h = h[~h.index.isin(l.index)] # drop index that appear in both
        l = l[~l.index.isin(h.index)] # drop index that appear in both

Then it's up to you what to do the resulting series, ie. you can use the indexes to slice your price data into swing segments or make a new series from h and l values and interpolate in between if you need just numbers.

Here's the example how this code worked on the price date of some undisclosed stock. The OHLC candlesticks were plotted in blue and the swing series in green. The swing series on the top subplot resulted after 1 iteration and the one on the bottom subplot resulted after 5 iterations:

The first iteration results stick very closely to the candlesticks, but the fifth iteration seem to capture the bigger swings. The accuracy is of course debatable, but in my opinion this method can be a good starting point for trend analysis. There is plenty of ways to improve it, ie.:

• add smoothing to the input High/Low price series;
• make sure that the results alternate from min to max to min, so there are no clusters of minima or maxima in the series like it occurred twice in the bottom chart of the example above;
• find ways to automatically adapt the number of iterations to achieve the needed results, etc...

Answer 2

You didn't say what you were going to do with the result and so there are two answers.

First, if you do not care about the future and the only concern is to precisely match the data that happened, then you can always fit any curve by constructing a polynomial whose degree is equal to the number of data points. It will fit perfectly. It will be useless as anything other than a descriptor of prior points, but it will look just right. It will be a case of overfitting. There is a nice example of overfitting the presidential elections at https://stats.stackexchange.com/questions/128616/whats-a-real-world-example-of-overfitting .

Your model is clearly overfitting the data for any predictive purpose, but very close to the exact solution if you are looking historically. The reason you can tell it is overfitting is that it is that it is catching the wavelets. The wavelets are known to be purely random and without signal. The original article on this can be found at:

Slutzky, Eugen. The Summation of Random Causes as the Source of Cyclic Processes. Econometrica. 5(2). Apr. 1937. pp.105-146

If you are trying to predict an outcome and intend to use it, then you need to adopt Bruno de Finetti's Coherence Principle to get there in your underlying mathematics. This will restrict you to Bayesian Statistics. Non-Bayesian methods are never coherent.

To understand why there is a simple 250-year-old example from the Reverand Thomas Bayes himself. In this example, the Reverend Bayes imagines a billiards table where a ball is bounced around to arrive at a random point. The ball splits the table into region A and region B randomly. The probability that a ball will roll into an area is equal to the proportion of area that region has divided by the total area.

The ball is removed after its location is recorded. Then two billiards players shoot balls. If the ball lands in their region they win and if not they lose. The problem is nobody told them where their region is or if they got a point or not until after it is over. They don't know where to shoot. The first to six points wins.

The question that needs to be solved is the odds that player B will win, given that eight shots have been made and the score is 5-3 in favor of A. A needs one point to win and B needs three.

The Frequentist answer is $(3/8)^3.$ This corresponds to gambling odds of about 18:1. So if you were to use a Frequentist method, then if you were a bookie you should offer 18:1 odds that B will win.

The Bayesian answer is very different. The Bayesian solution asks two different questions. The first question is what do we know about the odds of winning before we saw the balls shot. Second, what is the set of probabilities that could have triggered the exact results that were seen and what are the odds that each of these probabilities is the one true probability?

First, the user of a Bayesian method would create a parameter $\theta$ and assign the probability $\theta=k,\forall{k}\in[0,1]$. Since all possible values are equally probable, the prior distribution of beliefs about $\theta$ should be that $\Pr(\theta=k)\propto{1},\forall\theta\in{[0,1]}$. There is a binomial likelihood, so its probability is $\theta^3(1-\theta)^5$.

The posterior density function is $504\theta^3(1-\theta)^5$. This is the summary of all possible parameters that could trigger a 5-3 position and the probability that this is the true parameter. We need a prediction, however. To do this we make a prediction over the entire density, which is to predict the probability of winning three out of three. The solution to this is to solve $$504\int_0^1\theta^3\theta^3(1-\theta)^5\mathrm{d}\theta.$$

The answer to this is $\frac{1}{11}$. The Bayesian solution would create 10:1 odds. The Bayesian would be correct and always would create the correct solution. The Bayesian solution is $E(p^3)$, while the Frequentist is $[E(p)]^3$. This is true for any Bayesian versus Frequentist process including regression.

You would use Bayesian model selection. You would create a model that said there were only one set of parameters such as $$\frac{x_{t+1}-\beta{x_t}-\alpha-\varepsilon_{t+1}}{\sigma}.$$ Then you would assume there was a break and you would model two series jointly, one right after the other. You would do this again for three, four and five and so on until you felt you didn't want to continue.

As you increase the number of possible breaks, Bayes theorem will begin penalizing you for the added structure. At the same time, since it fits better, Bayes theorem will reward you for the improved goodness of fit. The model where the penalties are small and the rewards are high will end up with being the best fit model. If you want to do a prediction, then you will use the Bayesian posterior predictive distribution as we did to calculate the odds.

Because there is no prebuilt tool for this, you will have to build your own. If you have not used a Bayesian method, begin with William Bolstad's Introduction to Bayesian Statistics, 3rd edition. Definitely get the third edition as it is a huge improvement.

Answer 3

Your question is very general and I am sure it can be approached from different angles. Segmenting a time series as per the diferent components may help you to forecast each individua part also segmenting it in time spads as per volatility as well, comparing these also. In this regard consider, (I did some minor thinking on this a while ago). I am sure other people may have better ideas or it can be approached from a technical analysis point of view which where there are a number of toos you can use from). This is what I though a while ago.

Making the series stationary by taking diferences

1. Detrending the series. (Long term trends). Once this is done. which you can use this to forecast it).
2. Applying a loess regression to extract cycles to detrended series (smaller trends) - careful consideration to variance and degree of the polynomial to apply. I do not remember well but time spand I think should be optized as per variance. (you can also use this as to forecast the cycle)
3. if you want you can segment the obtained series as per regime change methods (usually mean and variance should be used).
4. Applying wavelets to the segments maybe usefull if there is a significant signal somewhere, however in general signal on stock indexes is minor (probably larger on individual stocks but I do not know how significant). You can apply wavelets (however they have boundary issues and levels which forbit forecasting) you can then use very short length filters and extract the higher frequency ones ( where most of the signal and noise resides ). In order to obtain a signal from noise you have to model the types of noises that you know or think you know exist in the signal and extracted from the signal (this is the trickiest part).
5. Apply some robust regresion models to forecast the signal and use volatility models to apply to noise (if mostly white noise just use vol models). (however regression is tricky - read some econometrics papers and talk to some persons in the field to get a sense of numbers and how to use it properly in regression). Some times I have been told using properly a regression may be much better than using a bunch of processes as explained).