Determining if a time series is random

Question

I originally posted this in the Data Science Stack Exchange. Another poster suggested I post it here.

The idea would be to identify "orderly" segments within a market time series and use them to inform a machine learning algorithm either by identifying relationships between orderly segments (or combinations thereof) or simply projecting price for some amount of time in the future. The rationale of doing it this way is that I believe humans look at charts in terms of vectors (i.e., length and net movement as described below) with varying degrees of certainty based on the linearity of the net movement. So this is an effort to apply machine power to human-style thinking.

---

The original post...

An example time series would be the stock market, which is sometimes described as a random walk. Over time, this is clearly not the case as it has essentially gone in one direction (up) with only occasional setbacks.

I'm coming up with several characteristics that could be used to measure orderliness of a specific segment of a time series: 1) Net movement from start to finish 2) Linearity of movement 3) Length of time

In short, the larger the absolute value of the movement relative to the standard deviation of the series over similar lengths of time and the more linear the move, the more significant that segment. These characteristics could also form a kind of signature of each given segment.

Maybe the answer to this is simply that a time series whose segments don't fit a normal distribution is a non-random time series, but I'm wondering if there are methods that capture the significance of a time segment relative to #1-3 above, i.e., "how rare is this segment"?

---

EDIT: Here's an attempt to visualize an intermediate goal of this strategy, to simplify a chart into "logical" segments like a human would do (using a logarithmic chart). The first chart would be an extremely insensitive method for grouping parts of the series and the second is a more sensitive version. Multiple analyses at various sensitivities could be performed simultaneously and used to train a deep learning algorithm where I believe patterns could be recognized similarly to how a deep neural net might recognize handwriting.

This strategy would probably only make sense in short intervals since fundamental forces would be more likely to prevail over longer horizons.

Answer 1

So there are several issues with your posting that you will need to resolve. The first one is your concept of randomness and distinguishing between a random event and a non-random event. To understand the problem, I think I should tell you a story.

You go home for a family reunion and see a tree you used to climb as a small child. You see the branch you used to sit on and decide to climb the tree. It looks so different from up there, maybe because you were so small and the world felt so different.

At the edge of the field your tree sits in, you hear a ruckus. The reason it is a ruckus is that I don’t get to use the word ruckus much. You see a squirrel running, being chased by a cat which, in turn, is being chased by a dog. The squirrel heads straight for your tree.

As the squirrel scampers up the tree, you realize that it is going to head for you, and the dog is directly underneath you, barking insanely. You sort of leap with shock when the squirrel climbs you to get away, and the cat jumps, claws extended onto you to catch the little rodent. The shaking and the added weight causes the branch to snap.

In the fall, your leg is broken; the dog is killed; and, the squirrel is captured and eaten once the startled cat realizes that it landed well while the squirrel went tumbling out of control. Was the event deterministic or random?

Surely, you would not have climbed the tree if you had predicted the future. The cat, the dog, and the squirrel were responding to instinct and had only one possible outcome path. Yet, was it random?

Indeed, the probability of that sequence of events was very low. Viewed as a random accident, it could have been a Farmer’s Insurance advertisement.

Since actuaries estimate precisely this type of risk, it can be treated as random. However, no physically random behavior happened. No coins were tossed. No random numbers were generated. No randomness occurred in the behavior. Each behavior could be deterministically be predicted with enough information.

Your argument that some behaviors are random, and some are not cannot hold. What you should instead think about is the presence of two distinct types of random behavior. One type of random behavior will not generate excess profits. The other type can generate excess profits.

Alternatively, you should assume there are two sets of totally deterministic behaviors. One set of actions produces results that cannot be predicted, and so no excess gains can be made from them. The other set is predictable, and excess profits exist.

Either of these ideas would work as the operationalize the same.

Your second problem is that returns are not normally distributed. Economists assumed normality or log-normality without any grounding. There is every reason to believe that the distributions involved lack a first moment. In this post "distributions involved" and others, I have argued that the distribution of going concerns if you strip out bankruptcy, merger, liquidity, and dividend effects is $$\left[\frac{\pi}{2}+\tan^{-1}\left(\frac{\mu}{\sigma}\right)\right]^{-1}\frac{\sigma}{\sigma^2+(r-\mu)^2}.$$

There are several difficulties with this distribution. The two most important, for your purposes, is that in logarithmic form the distribution lacks a covariance matrix so you cannot do something like ordinary least squares, and that the distribution lacks a sufficient statistic. For your purposes, the lack of a sufficient statistic means you either must use highly efficient Bayesian methods or relatively inefficient non-parametric methods such as Thiel's regression or quantile regression.

The third problem depends on the frequency of your trades. If you are using only end-of-day trades, then everything in this paragraph is irrelevant to you. Otherwise, it matters.

The ticker or the tape, whichever way you would prefer to designate the time stream of trades isn't literal. The trades on the tape are not in the order in which they happened. Likewise, the market high and low prices may not be the true high and low. The issue is and has always been that some trades happen "off the tape."

If a trade could shift a market price, the trade is taken off the tape and hidden until later. Consider a thinly traded stock with an order to purchase 10,000 shares. That large of an order would cause a large shift in the price, especially if people such as day traders or technical traders were aware of it. That trade is legally and purposefully hidden in the background.

If, over the course of the day, the trade accumulates all its shares, then those individual trades will never be reported. Instead, it will appear as a single trade of 10,000 shares at the weighted average price. If the day's high or low were part of that trade, then those prices wouldn't be the ones reported. Instead, the highest and lowest trade observed on the tape is reported. Also, the block order trade is often delayed until such time as the fact the information about it would no longer impact the market.

The fourth problem is created by a problem called coherence. A statistical methodology is coherent if fair gambles could be placed on it. Bayesian methods are coherent. Standard methods like ordinary least squares regression are almost never coherent. Neither quantile nor Theil's regression are coherent methods. You can find a pattern but the estimators are too inaccurate to use. Someone like a Warren Buffett could rig a game against you and take all your money.

Bayesian methods are advanced methods and require the use of calculus.

The distribution above is wide. If you used it in regression, the central ninety-nine percent of the mass is $63.657\times{2\sigma}$ wide, not $6\sigma$ as would be the case with a normal distribution. You need a remarkably good model to get significance because the natural noise is incredible.

Finally, any gains will likely come from liquidity effects. I ignored them above because they require advanced modeling. I would recommend reading Ashok Abbott's chapter in The Valuation Handbook. It is quite good.

To summarize:

1. The distribution above lacks a standard deviation. Neither it nor the mean exist so you cannot use them for anything.
2. If you use a non-Bayesian method, then someone like me may steal your money.
3. You have a restricted toolset; you will need to learn advanced methods.
4. Under the logarithmic transformation, no two sets of returns will covary though they will not be independent either. You cannot easily gather information about one security from another.
5. The list of intraday trades is neither in the order nor the magnitude that they occurred in. Apparent intraday patterns are intrinsically spurious because the order of events has been partially hidden and partially scrambled.
6. The noise in the system is tremendous.
7. You need to describe a two or more state scheme where you could profit from excess movements.

So, as a background to the following here is your answer.

1) Net movement from start to finish 2) Linearity of movement 3) Length of time

For 1, all the issues above apply.

For 2, a linear difference equation is equivalent to an exponentially growing continuous-time process. Linearity, by itself, isn't enough for your purposes as a test.

For 3, because trades are "off the tape" you cannot know how much time or when events actually happened, except for shares that have thin trading on that day. Unfortunately, the only way to know if there was thin trading is after the fact because block trades would be hidden until after you needed to know they were happening. So, while end-of-day trades are valid as to price and to some extent approximately correct as to volume since some blocks will finish near the end, pattern hunting by time to complete of a pattern won't help because the data has been partially hidden and partially scrambled.

PERSONAL NOTE Personally, as a former professional investor and as an academic researcher, I believe that a limited amount of technical trading is valid.

However, I think it is like surfing. You cannot surf almost anywhere in the world. Surfing requires a range of rare conditions. I suspect those conditions exist and that the market maker allows them to exist as a risk mitigation scheme. I have never tested those ideas as I have bigger fish to fry.

EDIT There is a reason that a reason that there is no textbook called "Statistical Handbook for Technical Analysis." There doesn't exist a simple solution. Even with neural networks, you would have to be very careful regarding your loss function.

Intraday content would be difficult because intraday data are not a time series as the items are not reported in the order of occurrence. You can rely on small trades for accurate reporting of information but nothing else.

If you are going to use a neural network, then you will need to add bankruptcy and mergers into your data set. They have different statistical properties. Just realize that everything obvious or simple has already been tried.