How can certain numerical distributions yield misleading standard deviation calculations? [closed]

What numerical distributions yield misleading standard deviation calculations? Can you make the standard deviation of distribution 1 attain a higher value than the standard deviation of distribution 2 where they are from two different discrete probability distributions?

I am aware that standard deviation is sensitive to the number of sample points.

Also, I am interested in the same question for downside standard deviation.

closed as unclear what you're asking by LocalVolatility, Bob Jansen♦Dec 9 '16 at 8:07

Please clarify your specific problem or add additional details to highlight exactly what you need. As it's currently written, it’s hard to tell exactly what you're asking. See the How to Ask page for help clarifying this question. If this question can be reworded to fit the rules in the help center, please edit the question.

• You need to define misleading and give an example. Also I don't quite follow your comment on distributions. – Bob Jansen Dec 9 '16 at 8:06

There are no distributions that have a standard deviation that trigger misleading calculations. Not all distributions have a standard deviation. The standard textbook estimator for variance is an unbiased estimator for variance, but it is a biased estimator for the standard deviation. This is because taking the square root of an expectation, which is what you are doing when you take the square root of the estimate of sample variance, is not necessarily equal to taking the expectation of the square root, which is what an unbiased estimator would do. This is known as Jensen's inequality.

Because of this you can sort of play small sample size games on purpose with a handful of discrete distributions that have been rigged to trigger this property, but that is more of a parlor trick.

Now among the continuous distributions, the distribution of returns for most assets lacks the property normally called variance. The US stock market returns for going concerns is close to a truncated Cauchy distribution, with the slight skew being due to the budget constraint.

A standard deviation can be thought of as a property, like a nose is a property of many animals, but absent in trees. If you saw something with a nose, you would not say it was a tree. Some distributions have no standard deviation and so any calculation of a standard deviation will be misleading as it will generate a random number instead. The symmetric Cauchy distribution has the peculiar property that as the sample size goes to infinity, the sample variance will go to infinity. Conversely, when you do this with a normal distribution you will get a near perfect estimate of the true variance.

Nonetheless, the only way to trigger this type of behavior in discrete distributions is to bin a distribution that has no variance. Binning will not cause a variance to appear, but it would be a weird thing to do with a discrete distribution, on purpose.

The symmetric Cauchy distribution is $$\frac{1}{\pi}\frac{\sigma}{\sigma^2+(x-\mu)^2}.$$ With a little bit of creativity and a good proper understanding of how to solve the constant of integration and evenly spaced bins, you could create a discrete distribution, say over the values of the Cauchy distribution on the integers, that would generate a misleading variance because it would not have one.

You can find a derivation for the returns on various asset classes at https://ssrn.com/abstract=2828744 It is technical however so I wouldn't worry about it too much. It is listed more for completeness. As your technical skills increase, you can come back to it.

• @use25459, Thank you for your superb answer. How do I modify the R language ATR function , Average True Range, to mimic downside standard deviation for lossess only? I am accepting your great answer now. – Frank Dec 9 '16 at 0:16
• Nice answer, glad you could do something with this. – Bob Jansen Dec 9 '16 at 8:08
• Frank, you cannot generally modify R functions as they are usually compiled in C or Python. Instead, you need to write your own script. The distribution of the daily high and the daily low should follow Gumbel distributions, but I do not think that anyone has solved summation or difference distributions for the Gumbel. I have solved the ratio distribution and I don't want to solve the difference distribution. My suggestion, if what you really want to know is downside risk, is to measure it as variance conditioned on the existence of a loss and skip ATR. – Dave Harris Dec 9 '16 at 17:11
• @user25459, Thank you for your very accurate comment. May I ask the best algorithm for measuring the moments of a symmetrical distribution of profits and losses? Thank you. – Francis Tuan Dec 10 '16 at 15:32
• There is a paper on that as well. There are no moments if you read the above paper. Derived from first principles, the distribution of returns for the Markowitz model should be $\frac{1}{\pi}\frac{\sigma}{\sigma^2+(r-\mu)^2}$. This distribution only has a zeroth moment, which, of course, is one. Your best estimator will be a Bayesian one because there is no sufficient statistic. You have to lose information otherwise. – Dave Harris Dec 10 '16 at 17:20