I want to ask you an advice about reading theory and examples of conditional expectation and conditional variance. I want to have my understanding deeper, because sometimes I can't understand calculation of this. For example, here: Expectation of $\frac {S_{T_2}} {S_{T_1}}$ at $T_0$ I always thought that Conditional Expectation is a random variable... Could you give me good links/books/articles?
Thanks. And cookies for all. :-)
Expectations and conditional expectations are either random or fixed points. It depends upon your choice of axioms. Unfortunately, I have never found a good book that fairly describes both well.
On the Bayesian side, I would suggest the very polemical "Probability Theory: The Language of Science" by E.T. Jaynes. On the null hypothesis side, I would suggest the undergraduate text "John Freund's Mathematical Statistics." The late John Freund wrote my introductory undergraduate text back when dinosaurs roamed the Earth. The undergraduate text for "Mathematical Statistics" as opposed to "Elementary Statistics" provides a good grounding in null hypothesis methods.
Most graduate students are trained in null hypothesis methodologies. In that axiomatic framework, parameters are fixed points and data is random. Because the null hypothesis fixes the parameter space all randomness is due to chance alone. The probability test is of a result as extreme or more extreme than the observed result. Randomness is chance.
Bayesian methods are orthogonal to null hypothesis methods. Parameters are random and data is fixed. After all, you saw the data, there is no uncertainty about it. It is fixed. The data fixes the sample space all randomness is due to uncertainty about the location of the parameter. The probability test is about the truth of a hypothesis given the observed data. Randomness is defined as uncertainty.
You will need to be mentally careful when reading books on either one as they often define the same words with fundamentally different meanings. The simple example is the definition of an expectation.
The expectation under null hypothesis thinking is $$E(\tilde{x})=\int_{\tilde{x}\in\chi}\tilde{x}p(\tilde{x})\mathrm{d}\tilde{x},$$ while the expectation under Bayesian thinking is $$E(\theta)=\int_{\theta\in\Theta}\theta{p}(\theta)\mathrm{d}\theta.$$
Using Keynesian notation, a Bayesian test of a hypothesis is $\Pr(\theta|X)$ while a Frequentist test is $\Pr(X|\theta)$.
Some terms, such as conditional probability, don't resemble the meaning in the other framework. All Bayesian inference is called conditional probability. A conditional expectation in a Bayesian framework is a posterior expectation $E(\theta|X)$ and is a random variable. An unconditional expectation would be a prior expectation $E(\theta)$.
On the null hypothesis side, it is a bit more complicated. An unconditional expectation is just the expectation of the distribution involved, $E(P_\theta(X))$. Conditional expectation is more complex. It depends on whether you are conditioning on a stochastic or non-stochastic variable. The added richness to the discussion comes from the differing role the sample space has. On the Bayesian side, all data is fixed and the remainder of the sample space is discarded as irrelevant.
As for links, on the null hypothesis side consider reading Deborah Mayo whose area is the philosophy of science. Her website is https://errorstatistics.com/
Alternatively, you could read Cosma Shalizi who is a statistician at http://www.stat.cmu.edu/~cshalizi/
On the Bayesian side, consider Andrew Gelman, a statistician, at http://www.stat.columbia.edu/~gelman/
or consider the psychologist Eric-Jan Wagenmakers at https://www.ejwagenmakers.com/
There is also a good posting on an existing stack exchange via the idea of an interval. The post constructs Frequentist confidence intervals for a data set of cookies versus the same Bayesian credible intervals (also called credible sets). It also gives a good idea of how the two groups think of conditioning. Since the intervals do not match and do not have the same properties it gives a way to think about the consequence of considering one thing random versus another. It is at https://stats.stackexchange.com/questions/2272/whats-the-difference-between-a-confidence-interval-and-a-credible-interval
