Long time viewer, but first time poster, so excuse me if i'm in the wrong place please.

Anyway, I am working on a project that is pretty interesting. Through data mining, I am able to gather a ton of investment portfolios. Each portfolio has the obviously related statistics, including total revenue, total loss, resulting profit, and I can even get a daily break down of this for the past 3-4 years. (1200 days exactly).

I did a bit of excel manipulation, using the r2 pearson function and using a sharpe ratio (average profit/standard deviation), and I think I am on the right track because I was able to get a few results that were pretty damn consistent. First I break up the 1200 days into 20 day segments, and I get the net daily profit/loss. Then I accumulate the 60 points of 20-day intervals to get a growing summation. When you do this, you'll get some sort of line since you're adding up the individual intervals, and then I apply the 2 functions of r2 pearson and sharpe ratio to this line. My goal is to find portfolios that are linear in fashion, because in my eyes, if the portfolio has been linear for the past 1200 days, it should have a high chance of continued linearity.

So when you sort out my list of portfolios by highest sharpe ratio, I can choose one of the top portfolios and as you can see from the pic of the graph attached, it is somewhat consistent and linear as opposed to another portfolio that, when graphed, appears to be sinusoidal, or erratic with large jumps and dips.

My question is, can anyone give me more information on how valid my theory of using linearity for continued success is? Is there a different equation that I should be using to determine which portfolio I can invest with in the future? Is there something more capable of defining linearity of my portfolio graphs than the r2 function or the sharpe ratio?

enter image description here

I was also thinking of using a portfolio that ranks high in =(average sum)/(loss)) or choosing a portfolio that has the least amount of losses in it. Typically though, I have observed those having alot of flat areas when graphed, and I'm not sure that would be best for future success.


Updated (1/25/2021) Below is a regression analysis done via excel using toolpak. I know the p-value is important, but is there anything in here that says this is terrible?enter image description here

  • 1
    $\begingroup$ Hi, welcome and sorry to be bearer of bad news, but there's a famous paper (Granger and Newbolt 1974) that was able to generate these kinds of results and outcomes, using purely random walks! The crux of the problem is any accumulated price-type series is inherently "non-stationary", because future (price) outcomes are inherently biased by past success of failure. If the daily/weekly/monthly returns are consistently positive, that might be good; but how they get there matters. "Linearity" could just be "spurious correlation" of price and time (as per G&N 1974). $\endgroup$
    – demully
    Commented Jan 25, 2021 at 1:30
  • $\begingroup$ Hey @demully thanks for responding. I'll take a look at G&N 1974 so I have a better understanding of it. I'm not sure if it matters, but the portfolios that I can select from are not directly reading price data. They are a conglomerate of variables and investments. I am simply given the results of all of the portfolios that I can invest in and make a choice. E.G. I was given a portfolio with a single investment strategy of a single security, and although the price data of the security was declining, the ROI from the portfolio was correct...the portfolio was profitable in entering and exiting $\endgroup$
    – nhb
    Commented Jan 25, 2021 at 2:02
  • $\begingroup$ Continuing on ^. I skimmed the G&N 1974, and I stumbled on the durbin-watson statistic. Do you think that would help me determine whether my data is above/below the durbin-watson acceptance threshold? If not, do you think there is a better approach that I can use, given the data that I have from the portfolios? Thank you for this by the way, I learned something new! $\endgroup$
    – nhb
    Commented Jan 25, 2021 at 2:04
  • $\begingroup$ Lord, you;'ve just expanded this one... needs a proper answer > comment. DW is useful; but not really for the underlying problem here. More relevant are "Dickey-Fuller" tests (and there are at least three of those out there ;-) statisticians love their tests ;-) Google them for starters, but be warned - they are "uncertain" as juries giving a verdict on limited and conflicted evidence. Nature of the beast., I'm afraid. $\endgroup$
    – demully
    Commented Jan 25, 2021 at 21:50
  • $\begingroup$ @demully haha yeah there is no doubt this gets complicated. I learned a bunch from reading up the G&N that you mentioned earlier...When I did a test run, my line graph failed the DW test, which was uspetting, but after further digging I think there is a way out. Technically, the DW test is for linear regression and mine actually is a time series... So I started on the path and sure enough, my line graph was ACCEPTABLE in the DW test for time-series. It also mentioned something called autoregressive AR1 model, which I have to look into. I'm sure Dickey-Fuller will be just an informative :) TY $\endgroup$
    – nhb
    Commented Jan 25, 2021 at 22:38

2 Answers 2


The problem with your question is phrase "continued success". Any formal test exploring linearity and upward direction would be based on the assumption that the future data are well-represented by the historical data. If this was the case, you could estimate model $$ \text{Portfolio} = \beta_0 + \beta_1 * \text{Day} + \beta_2 * \text{Day}^2 + \varepsilon. $$ Then you could use bootstrap sampling to show that

  • $\beta_1$ is statistically significant and positive (upward direction),
  • $\beta_2$ is non-significant (linearity over time).

The problem, of course, is that the past is not indicative of the future. Even if some crafty trading strategy is behind the overall portfolio performance, sooner or later this strategy is discovered or becomes less effective.

  • $\begingroup$ I think I read this last night. check out this link: online.stat.psu.edu/stat501/lesson/14/14.1... autoregressive modeling. I will apply it and I'll see how it turns out. Maybe I can test it with a previous set of data and check out the results that WOULD HAVE happened. But I do have a question with this. From what I understand, AR1 is first order, so that means that ONLY the preceding value is used to predict the next. However, Intuitively you can assume this will be inaccurate. I would assume the more data -> the more accurate. So in my case, should I use an AR(60) ? $\endgroup$
    – nhb
    Commented Jan 25, 2021 at 22:42
  • $\begingroup$ The minimum-variance portfolio has been known for half a century but has not become less effective. Although new strategies have appeared that slightly outperform it, they have not made it any less effective of a strategy $\endgroup$
    – develarist
    Commented Jan 26, 2021 at 22:29
  • $\begingroup$ So I'm assuming the general consensus is that strategies/investment plans are good up until they become "public" or more widely used, and if that happens then they become less effective for the individual because there's a ton of buying/selling based on the same entry/exits, correct? But you're also saying that min variance, (Return/std Dev) is relatively solid. I definitely agree, but determining when that trend will end... either when the std dev rises or the return decreases... that's what I'd like to protect myself from $\endgroup$
    – nhb
    Commented Jan 27, 2021 at 0:36

This won't be a brief one...

The potential pitfall with linearity (or log-linerarity) is best explained with two examples. There is a single (US equity) fund manager who was able to beat the S&P500 index sixteen years in a row. He was called Bill Miller; and he used to manage the LeggMason US Value fund. There aren't or have never been (given frequent replacement of failures by new attemptees) 2^16 fund managers (~65,000) out there to suggest this would be likely. But this isn't the point. The punchline is that Bill retired in 2009, after having just given up all of his accumulated 16 years of steady outperformance in 18 months through the GFC. He was "linear up" before a structural break down.

A less anecdotal version of the same would be consider the returns of call-overwriting strategies. Most of the time, these will enjoy a greater share of underlying market returns than the loss in market exposure, giving you more of the market upside for less of the volatility. The straight-line up appears cleaner. Except the saw-tooths down (equal and opposite in risk-adjusted terms most of the time) appear equally and oppositely more jagged. Hedge funds habitually have returns that share similar statistical properties. So the question is how much anyone's samples of returns contain more or less than the population of either regime.

But let's park such cynicism aside. We are looking at a time series in which we can be almost-certainly confident that it's non-randomly appreciating, even adjusting for any reasonable numeraire (like inflation and/or dividends etc.). It stands to reason that the more linear (or log-linear) it is, the better a standard regression fit would be. So the lower the standard error, the greater the precision around that fit, the greater the confidence in predicting its parameters.

Except here is where the rub lies. There are multiple different ways to create the charts you are looking at, each of which has different statistical properties, that allows you to infer different conclusions (even assuming the past continues into the future). It could be a trend (which is what regression analysis assumes). It might be a random walk with drift; or an "almost-random walk", ie an autoregressive process. It might even be a proper random walk with once-in-lifetime good luck ;-)

These can all appear very very similar; but have very different investment implications if that is the true process. And the barrage of statistical tests out there are all doing their best; but they do struggle to distinguish between Tweedle-Dee and Tweedle-Dum above.

The Durbin-Watson (DW) is a basic test for autocorrelation. But there is also eg Breusch-Godfrey, which doesn't assume (like DW) that any datapoint has to be related to its immediate neighbours. What this family of autocorrelation tests do is tell you that any deviations of datapoints from the regression line cease to be random. In which case, linearity ceases to be relevant or representative; and the usual assumptions and conclusions associated with regression cease to apply.

But hope is not lost. Your attractive time-series, if not linear, might be an autoregressive process, a random walk with drift, or an extremely lucky drunk just flipping heads over a good year or decade. The critical distinction between these is the existence of a so-called "unit root", ie is the beta of today's price =1 or <1 to yesterday's price. Which is what the Dickey-Fuller tests attempt to work out.

If it's autoregressive (ie beta to prior >0 as per DM, but <1 as DF), then you need an AR() model that includes the prior as an additional correcting input to your linear or log-linear model. Then the residuals should be random (or a million miles closer to such), and regression on these terms remains a valid best-guess to reality.

But if it's a random walk with drift, then you cannot regress; but are stuck in the classic mean-variance of Markowitz et al. A high mean with a low vol would remain attractive; but the trade-off is a risk-reward trade-off; and any kind of regression on price would be superfluous at best and misleading at worst.

The caution about linearity in general is that is you assume the analytical worst case, that there is no linearity, not even an autogression, and it is then just a random walk, then the analytical path of least resistance would be to calculate the Sharpe Ratio. About which a ton of papers have been critical in the extreme as a measure of risk.

It's hard to win in this quant game, really. Quant finance is a pick yourself out of the gutter, reboot, and try again, again, and again game like no other ;-)


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.