How random are financial data series?

Question

Pseudorandom number generators are often tested using e.g. a test suite like Diehard tests or Dieharder. If one would run these tests e.g. on stock market time series or other financial data, would you expect your financial data to qualify as being a good random number generator or would it fail in many of these tests?

Answer 1

Variance ratio tests have been used numerous times to show that financial asset prices do not follow a random walk. You can for example look at

-Lo and MacKinlay : Stock market prices do not follow a random walk : http://press.princeton.edu/books/lo/chapt2.pdf (US Stocks)

-Hoque, Kim, Pyun: A comparison of variance ratio tests of random walk: A case of Asian emerging stock markets.

From a quick search, I haven't found any article using the kind of test you talk about. It seems like a new interesting approach so I completly agree with Dirk.

Furthermore, it would be interesting to change the time frame as asset can behave quite differently on different time frames. Liquidity should also be a factor.

Please keep us posted.

Answer 2

Why don't you try it and report back?

Recall, though, that while a random walk is often a rather competitive forecast, realized data is understood to have weak dependence especially in higher moments.

Having worked a bit with DieHarder, I'd suspect it to reject a number of series. But the proof is in the pudding...

Answer 3

I have tested lots of forex data for randomness. Some currency pairs are very close to random walk. And the problem is open question, because there is no uniform explanation what the random walk is. According to Mandelbrot, Taleb and some other authors randomness can be different. Even if the data is not random it doesn't mean it can be effectively traded. In other words, test for existence of trade opportunity is more useful than randomness test.

Answer 4

All the ideas above are great ideas.

Another kind of test would be an idea borrowed from Random Matrix Theory.

Assemble your time-series into a matrix. Evaluate the distribution of the eigenvalues of the matrix vs. the distribution of a random matrix. Turns out that the distribution of eigenvalues in a random matrix conforms to distributions such as the Marchenko-Pastur distribution.

If the distribution of eigenvalues conform to the eigenvalues predicted by a random matrix (where each entry corresponds to a draw from a standard normal random variable, for example), then there is probably not information in the time-series. Here is an illustration of Wigner's semi-circle law for a random matrix.

Answer 5

would you expect your financial data to qualify as being a good random number generator

Financial time-series, specifically price-change series, would make terrible random number generators because they generally contain significant dependencies.

or would it fail in many of these tests?

If you test for randomness, meaning, initial conditions do not completely determine the subsequent values, you will generally find price-change series to be random. I say generally because you will also find situtations (which come and go, and are rare) where price-change series are measurably deterministic.

If you test for dependence, you will generally find serious serial dependencies in price-change series. For example, a price increase followed by a price increase is generally far more probable than one finds with artificially generated random, independent price-change series.

I say "generally" because the statistical characteristics can change with time and with market, back and forth.

So, to summarize, the fact that price-change series are normally dependent is why technical analysis works. The fact that the series are random is why many smart people are fooled into thinking technical analysis doesn't work.

Answer 6

There are also the NIST tests used to examine the random number generators used in cryptographic systems. http://csrc.nist.gov/groups/ST/toolkit/rng/index.html

I played around with it a little bit, here is the first test, the monobit test.

library(stats)
library(quantmod)

code_input <- function(sym, fn=Cl) {
    return(na.omit(as.vector(ifelse(ROC(fn(sym)) >= 0, 1, -1))))
}

#from help(stats)
erfc <- function(x) 2 * pnorm(x * sqrt(2), lower = FALSE)


monobit <- function(v) { 
    sobs <- abs(sum(v))/sqrt(length(v)) ; 
    return(erfc(sobs/sqrt(2)))
}

run_test_block <- function(input, n=100) {
    nrand <- 0
    nnonrand <- 0
    cnt <- 0
    for (i in seq(1, length(input), n)) {
        if ((i + n) < length(input)) {
            cnt <- cnt + 1
            pval <- monobit(input[i:(i+n)])
            if (pval < 0.01) {
                nnonrand <- nnonrand + 1
            } else {
                nrand <- nrand + 1          
            }
        }
    }
    sprintf("%d tests %d random, %d nonrandom", cnt, nrand, nnonrand)
}

run_test <- function(input, n=100) {

    nrand <- 0
    nnonrand <- 0
    cnt <- 0
    for (i in 1:(length(input) - n)) {
        cnt <- cnt + 1
        pval <- monobit(input[i:(i+n)])
        if (pval < 0.01) {
            nnonrand <- nnonrand + 1
        } else {
            nrand <- nrand + 1          
        }
    }

    sprintf("%d tests %d random, %d nonrandom", cnt, nrand, nnonrand)

}

getSymbols("^GSPC", from="1970-01-01")
run_test(code_input(GSPC))
#[1] "10491 tests 10143 random, 348 nonrandom"
run_test_block(code_input(GSPC))
#[1] "105 tests 101 random, 4 nonrandom"

Make of that what you will. The NIST paper recommended that only the cases determined to be random by the monobit test be subject to the additional tests.

Answer 7

I stumpled upon a blog post where someone did this work and reported about it here: http://www.turingfinance.com/hacking-the-random-walk-hypothesis/

Some quotes:

The scores for the data sets lie between the scores of the two benchmarks meaning that markets are less random than a Mersenne twister and more random than a SIN function, but still not random.

Assuming randomness is not binary, one could conclude that not all markets are made equally "random". Some of the markets, namely the foreign exchange rate between the USD and GBP currencies and the S&P 500 Index, exhibit much lower levels of randomness than others such as the Hang Seng Index.

certain window sizes cause markets to appear less random. This may indicate the presence of cyclical non-random behaviours in the markets e.g. regimes.

in the presence of new or additional information (e.g. fundamental or economic data) the apparent randomness of the market may break down