# Memory effect of log returns of S&P 500

I am trying to reproduce the analysis discussed in https://arxiv.org/pdf/cond-mat/9905305.pdf where they use high-frequency data (1-minute frequency) of S&P500 from 1984 to 1996. In particular, they found the log returns for this period follow an exponential decay behaviour for the first 20 minutes with a characteristic decay time of 4 minutes, see figure below (that is the Fig.3 of the paper)

I don't have the data of the period from 1984 to 1996, but I have downloaded data from 2010 to 2021 (here the link) and I have tried to perform the same analysis with these data, but I find a completely different behaviour (see Figure below).

Question: Why am I not seeing the exponential decay? Do I really need more data?

The python code that I use to generate this plot is the following:

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt

def multiplyLag(series,lag):
return series * series.shift(periods=lag)

df=df.reset_index()
df['time'] = pd.to_datetime(df['time'],utc=True)
df=df.set_index('time')
df=df.sort_values('time')

logDataSet=np.log(df)
logDataSetDiff = -logDataSet.diff(periods=-1)
denominator = multiplyLag(logDataSetDiff,0).mean()['4']-((logDataSetDiff).mean())['4']**2
y=[]
for i in range(1,50):
print(i,end='\r')
y.append(1/denominator*(multiplyLag(logDataSetDiff,i).mean()['4'] - ((logDataSetDiff).mean())['4']**2))

x=range(1,len(y)+1)
plt.scatter(x, y)
plt.show()


and the file SPX.csv that I am using can be downloaded here for convenience.

• "Why am I not seeing the exponential decay?" - why would you expect to see the same behaviour decades later? Market structure has changed (e.g. HFT replaced specialists) and presumably any effect that could have been profitably exploited would have been (e.g. short-term mean reversion or momentum). Commented Dec 20, 2021 at 18:05
• @user42108 that’s true, that’s the first explanation that came out in my mind, but for some other reason I was believing that the macro properties of the two point correlation function shouldn’t have changed dramatically. Do you have any reference where the absence of such effect is discussed in modern data? Commented Dec 20, 2021 at 18:46
• For example, I think that I should see the same exponential decay if I had access to very high-frequency data. The point is that I don’t know if these data should be sampled at the 1second or 1millisecond scale. Do you have any reference? Commented Dec 20, 2021 at 19:09

## You will not be able to replicate positive decaying autocorrelations in current markets:

Due to the increased popularity in high-frequency trading, there's been a likewise increased focus on studying the underlying empirical properties of high-frequency data. One of the stylized facts in high-frequency data is the significant first-order negative autocorrelation of the log-returns — which you have rightfully captured in your Python generated autocorrelation plot — that is said to arise from non-synchronous trading and bid-ask bouncing, as seen from the perspective of Econometricians [1] [2] [3].

There's been a good amount of literature describing this fact for the past 25 years:

Therefore your second depicted plot, is more in line with current empirical literature than the study you're trying to replicate. Also, increasing the time-horizon or frequency of your current data will not increase the chances of producing positive decaying autocorrelations.

## Why don't you experience the same decay of positive autocorrelations?

• Different data-cleaning procedure: Even if you were to obtain SPX minute data from 1985 via your own data-vendor, you could still end up with slightly different results, since your data-vendor might use different cleaning rules, than the ones used by the authors (or their corresponding data-vendors).

• The underlying characteristics has changed (as also described by the above comment): The underlying characteristics of the SPX index has changed since the 1990s, which can be attributed to high-frequency trading becoming more predominant in todays markets. Moreover, the constituents of the S&P 500 index have likely changed within this time-span.

One of the cited papers on high-frequency returns exhibiting positive autocorrelations was Cutler et al. (1991). They showed how autocorrelations became more positive as the sampling frequency increased, even though the authors never used higher frequencies than a daily sampling rate. However, they specifically argue that different time-horizons and data constructions can affect the sign of the autocorrelations:

While these findings appear in many markets, they are not universal. Schwert (1989), for example, finds negative serial correlation in daily U.S. stock returns before 1917. An obvious issue for future research is whether differences in data construction, or variation in market structure through time or across markets, can explain some of the differences in the stochastic properties of returns.

In conclusion, it is unlikely that you would get the same results using the minute-returns of SPX from 2010 and beyond.

The best way for you to reproduce the results of the article: Ask the authors for the source of their data and if they have used a cleaning procedure. Or buy/get pre-cleaned SPX data in the time interval 1984 to 1996 from your own data-vendor and try to reproduce the results.

$$\:$$ Articles commenting on the first-order negative autocorrelation:

[1]: Andersen, T. G., & Bollerslev, T. (1997). Heterogeneous information arrivals and return volatility dynamics: Uncovering the long‐run in high frequency returns. The journal of Finance, 52(3), 975-1005.

[2]: Koutmos, G. (1997). Feedback trading and the autocorrelation pattern of stock returns: further empirical evidence. Journal of international money and finance, 16(4), 625-636.

[3]: Roll, R. (1984). A simple implicit measure of the effective bid‐ask spread in an efficient market. The Journal of finance, 39(4), 1127-1139.

• Thank you so much for all these references, I will have a look at them. Can you tell me if they are using the fastest available data? For example, if HF traders operate at the scale of milliseconds, do they study autocorrelations for log returns sampled at the millisecond scale? I will read the paper carefully though. I have a theoretical physics background and just recently I have been finding interest in this field. Commented Dec 22, 2021 at 13:08
• @apt45 In the papers, the highest available calendar-time sampling (CTS) scheme is a 1-second frequency. Be aware, that there exists different sampling methods (see Hautsch & Podolskij article for summary), with CTS being most popular. Wrt. CTS, there has been discussions that milisecond frequencies (even for very liquid assets) contains too much noise and thus produces a low signal-to-noise ratio, in order to be used within a predictive modeling setup. Also the marginal increase in performance when using milisecond-frequencies gets outweighted by the increased complexity in the models.
– Pleb
Commented Dec 22, 2021 at 13:54
• very useful information. Thank you very much! BTW If you are aware of any reference for modeling financial data using classical field-theory techniques, please let me know. Commented Dec 22, 2021 at 14:24