The distribution of p-values is uniform only when the null hypothesis is true and all assumptions are met. The difficulty with p-values is that they are composite functions.
Finding that an empirical set of p-values is far from uniform tells you one of two things.
The first is that your null is false.
The probability assumptions behind the model are ...