1
$\begingroup$

I've been applying the probability integral transform as shown here to standardise date for input into a neural network:

https://math.stackexchange.com/questions/592076/mapping-cdfs-to-each-other?noredirect=1&lq=1

I want to standardise the data that is wiebull distibuted by mapping it to standard normal:

The original histogram is shown below:

Original Histogram

When I transform to the standard Normal I get the histogram shown below, which doesn't look very Normal. I don't want the big gap I want the transformed histogram to look as much like a standard normal as possible. Is there anything else I can do?

Transformed Data

Here is the pdf of the inverse CDF it is approximately flat, When I normalize by the data set size the resultant sum is approximately 0.5? I would expect it to be approximately one? Should I be normalising the data by some other value other than the size of the data set?

The PDF of the Inverse CDF

Improve this question
$\endgroup$
8
  • 1
    $\begingroup$ Does not really look Weibull-distributed to me. Can you post the original data. If I understand well, calling it $X$, the second plot corresponds to $Y = \text{normcdf}^{-1}(\text{weibullcdf}(X))$ ? How did you determine the parameter of the Weibull distribution, MLE ? $\endgroup$
    – Quantuple
    Oct 5, 2016 at 9:10
  • $\begingroup$ The data is not discrete, (even if it was I's approximate using a continuous distribution so that I could apply the probability integral transform). The min value of 0.1 and max 23. It corresponds to a time series of financial data (bid ask spreads) these are not truncated per se but as they refer to tick data there is some discreteness to the moves. Weibull was best fit but I've tried several candidate distributions and they all produce a mapping qualitatively similar to this. I'm trying to convert to normailty as it will be input to a neural network and I need to standardize the inputs $\endgroup$
    – Bazman
    Oct 5, 2016 at 10:57
  • 1
    $\begingroup$ IMHO you should not use Weibull. Use the empirical CDF (for example, given the sample $X$ find the pdf $f(x)$ using kernels and integrate it), denote it by $F(x)$. Check that $F(X)$ is distributed as a $U[0,1]$. Then wirte $Y=norminv(F(X))$ to get your standard normal.... I do not see the point in doing that though... You're basically replacing your input by a standard normal. Why don't you just standardise by doing: $Y = (X-\text{mean}(X))/\text{std}(X)$ ? $\endgroup$
    – Quantuple
    Oct 5, 2016 at 11:49
  • 1
    $\begingroup$ All I meant to say is that maybe trying to fit a Weibull distribution, with its limited number of parameters hence limited degrees of freedom is not the way to go. You'll be able to fit it "optimally" (in the max likelihood sense for instance), but it will never be as good as using a non-parameteric distribution. Take this example, your data is chi-squared distributed. What is better, fitting a Weibull through MLE or using a non-parametric estimation? It's the same problem here, you don't know the true data generating process so why bother? Anyways testing $F(X)\sim U[0,1]$ should confirm. $\endgroup$
    – Quantuple
    Oct 5, 2016 at 14:36
  • 1
    $\begingroup$ Said otherwise, the answer you refer to in your post works because $X$ has exactly $F_1(x)$ for cdf. In your case, you do not know $F_1(.)$. You should therefore estimate it as best as possible given the information you have. Assuming a Weibull distribution is restrictive. $\endgroup$
    – Quantuple
    Oct 5, 2016 at 14:41

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.