Say for a stock I want to do a simulation using 30 days of historical returns, and maybe generate 1000 paths, with 2 days as the forecast horizon. Say I have 100 of these 5 day blocks used for generating the distribution, matched with the actual values of the 2 days I am interested in forecasting. How would I estimate the accuracy of this method of generating a distribution? And what is a method I could use to compare it with a different means of forecasting the distribution? Any suggestions would help.
Seems like you are interested in the forecast error
With regard to distribution, I would also look at the mean forecast error. A good model with have a mean error = 0, since it is not bias upward or downward.
You might want to perform a Two sample Kolmogorov-Smirnov test on the empirical CDFs of the distributions: your forecast and the realized distribution. Then use the p-value of the test as the metric of interest. Other statistical tests of difference in distribution which can be so abused are the Baumgartner, Weiss, Schindler test, and the variations thereupon. I believe there is also a 2-sample Anderson-Darling type test. There are also the non-symmetric Kullback Leibler divergence, and a half-dozen other definitions of statistical distance. Probably you are most limited by the statistical package or programming language you are working with. If using R, I would guess you can find implementations of all of the above.