I am looking to passively/lump-sum invest stocks. If I averaged the yearly returns of a stock, how many years would I need before I could say with 95% confidence that the averaged value would accurately predict the average yearly return for the next 10 years? If the answer is that "it depends", what factors would it depend on?
It seems implicit in the question that you are happy to assume that the distribution of historical returns is an unbiased and consistent estimator of the distribution of future returns. Else "it depends" (on every variable that affects stock prices, ie all of them) ;-)
You can always come up with such an estimate; but your confidence interval will just be absurdly wide in small sample sizes. Let's say you have 9 years of data, suggesting a mean of say 5% and a vol of 15%. The standard error is your vol divided by root-n = 15% / 3 = 5%. So you can be 95% confident (ie 2 sigma) that the average future return should be between -5% and +15%. Not exactly helpful; and 5% of the time the long-term future should indeed be outside this range!
Broad rule of thumb is you need to quadruple the sample size to halve the confidence bands. So 36 years of data would be +/-5% (ie 2 * 15%/6) around mean 5%, ie somewhere between 0% to 10%. Still not very helpful.
And you would need 144 years to pin this down to say +2.5% to +7.5% - assuming the 15% vol is correct, but that's the long-run average for the US stockmarket. For single companies (more like 25% vols), you would need maybe 100 years of data to be 95% confident the mean was 0-10% (ie 2 * 25%/10 = 5% standard error = same 5% mean).
But once you've been patient enough to find and gather all this data to give you these kinds of levels of confidence, you start to run into an inevitable problem. Is the stockmarket, let alone any company, I'm buying today really the same beast as it was 10, 25, 50 or 100 years ago? That's the real problem here.