The risk-neutral measure $\mathbb{Q}$ is a mathematical construct which stems from the law of one price, also known as the principle of no riskless arbitrage and which you may already have heard of in the following terms: "there is no free lunch in financial markets".
This law is at the heart of securities' relative valuation, see this very nice paper by Emmanuel Derman ("Metaphors, Models & Theories", 2011) and some part of this discussion.
In what follows, assume for the sake of simplicity
- existence of a risk-free asset ;
- deterministic and constant rates, with risk-free rate $r$ ;
- no dividends and no additional equity funding costs.
How to relate $\mathbb{Q}$ to $\mathbb{P}$: some useful concepts
The risk-neutral measure $\mathbb{Q}$ is a probability measure which is equivalent to $\mathbb{P}$ and under which the prices of assets (I should rather say the price of self-financing portfolios composed of marketed securities to be perfectly rigorous), discounted at the risk-free rate, turn out to be martingales.
If one assumes there is no free lunch in the real world (hence under $\mathbb{P}$), then the above definition (more specifically the "equivalent" part) suggests that there will be no free lunch under $\mathbb{Q}$ either. To convince yourself have a look at the accepted answer to this SE question. This answers your question concerning no arbitrage conditions.
The martingale property is convenient since it allows us to represent asset prices as expectations conditional on the information we currenty have, which seems intuitive and natural. Indeed from the definition if $X_t$ is a $\mathbb{Q}$-martingale then
$$ X_0 = E^{\mathbb{Q}}[X_t \vert \mathcal{F}_0] $$
The adjective risk-neutral comes from the fact that, using a replication argument (static for linear contracts, dynamic for most of the others) and under the assumption of no free lunch (+ market completeness, continuous trading, no frictions), one can show that the true performance of the stock simply disappears from the option valuation problem. Risk aversion thus disappears and only the risk-free rate $r$ remains. This is exactly what Black-Scholes-Merton showed and which earned them the Nobel prize in the first place, see below.
A simple example: the Black-Scholes model
Assume that the stock price $S_t$ follows a GBM under $\mathbb{P}$
$$ \frac{dS_t}{S_t} = \mu dt + \sigma dW_t^{\mathbb{P}}\ \ \ (1) $$
where $\mu$ is the expected performance of the stock and $\sigma$ the annualised volatility of log-returns. This equation describes the dynamics of the stock in the real world.
Consider the pricing (we are still under in the real world) of a contingent claim $V_t = V(t,S_t)$ of which the only thing we know is that it pays out $\phi(S_T)$ to its holder when $t=T$ (generic European option). Now, consider the following self-financing portfolio:
$$\Pi_t = V_t - \alpha S_t$$
Using Itô's lemma along with the self-financing property yields:
\begin{align}
d\Pi_t &= dV_t - \alpha dS_t \\
&= \left( \frac{\partial V}{\partial t} + \frac{1}{2} \sigma^2 S_t^2 \frac{\partial^2 V}{\partial S^2}\right) dt + \left( \frac{\partial V}{\partial S} - \alpha \right) dS_t\\
\end{align}
The original argument of Black-Scholes-Merton is then that, if we can dynamically rebalance the portfolio $\Pi_t$ so that the number of shares held is continuously adjusted to be equal to $\alpha = \frac{\partial V}{\partial S}$, then the portfolio $\Pi_t$ would drift at a deterministic rate which, by absence of arbitrage opportunity, should match the risk-free rate.
Writing this as $d\Pi_t = \Pi_t r dt$ and remembering that we've picked $\alpha = \frac{\partial V}{\partial S}$ to reach this conclusion, we have
\begin{align}
&d\Pi_t = \Pi_t r dt \\
\iff& \left( \frac{\partial V}{\partial t} + \frac{1}{2} \sigma^2 S_t^2 \frac{\partial^2 V}{\partial S^2} \sigma^2 \right) dt = \left( V_t - \frac{\partial V}{\partial S} S_t \right) r dt \\
\iff& \frac{\partial V}{\partial t}(t,S) + r S \frac{\partial V}{\partial S}(t,S) + \frac{1}{2} \sigma^2 S^2 \frac{\partial^2 V}{\partial S^2}(t,S) - rV(t,S) = 0
\end{align}
which is the famous Black-Scholes pricing equation. Now, the Feynmann-Kac theorem tells us that the solution to the above PDE can be computed as:
$$ V_0 = E^\mathbb{Q}[ e^{-rT} \phi(S_T) \vert \mathcal{F}_0 ] $$
where under a certain measure $\mathbb{Q}$
$$ \frac{dS_t}{S_t} = r dt + \sigma dW_t^{\mathbb{Q}} $$
which shows that
$$\frac{V_t}{B_t} \text{ and } \frac{S_t}{B_t} \text{ are } \mathbb{Q}\text{-martingales}$$
with $B_t = e^{rt}$ representing the value of the risk-free asset we mentioned in the introduction. Notice how $\mu$ has completely disappeared from the pricing equation.
Because this Feynman-Kac formula very much resembles a magical trick, let us zoom in on the change of measure from a more mathematical perspective (the above was indeed the financial argument... at least for deriving the pricing equation, not for expressing its solution in martingale form).
Starting from $(1)$, let us define the quantity $\lambda$ as the excess-return over the risk-free rate of our stock, expressed in volatility units (ie its Sharpe ratio):
$$ \lambda = \frac{\mu - r}{\sigma} $$
Plugging this into $(1)$ gives:
$$ \frac{dS_t}{S_t} = r dt + \sigma (dW_t^{\mathbb{P}} + \lambda dt) $$
Now Girsanov theorem tells us that if we define the Radon-Nikodym of the change of measure as
$$ \left. \frac{d\mathbb{Q}}{d\mathbb{P}} \right\vert_{\mathcal{F}_t} = \mathcal{E}(-\lambda W_t^{\mathbb{P}}) $$
then the process
$$ W_t^{\mathbb{Q}} := W_t^{\mathbb{P}} - \langle W^{\mathbb{P}}, -\lambda W^{\mathbb{P}} \rangle_t = W_t^{\mathbb{P}} + \lambda t $$
will emerge as a $\mathbb{Q}$-Brownian motion, hence we can write:
$$ \frac{dS_t}{S_t} = r dt + \sigma dW_t^{\mathbb{Q}} $$
Okay, this might seem even more magic to you than earlier, but there is a rigorous mathematical treatment behind don't worry.
Anyway, an interesting feature of writing and manipulating the Radon-Nikodym derivative is that one can eventually show that:
$$V_0 = E^{\mathbb{Q}} \left[ \left. \frac{V_T}{B_T} \right\vert \mathcal{F}_0 \right] = E^{\mathbb{P}} \left[ \left. \frac{V_T}{B_T} \mathcal{E}(-\lambda W_T^\mathbb{P}) \right\vert \mathcal{F}_0 \right]$$
where I have used the Bayes' rule for condition expectations, with
$$ X := V_T/B_T,\ \ \ f := \frac{d\mathbb{Q}}{d\mathbb{P}} \vert \mathcal {F}_T = \mathcal{E}(-\lambda W_T^{\mathbb{P}}),\ \ \ E^\mathbb{P}[f \vert \mathcal{F}_0 ] = 1 $$
The above result is extremely interesting and can here be re-expressed as
$$ V_0 = E^{\mathbb{Q}} \left[ e^{-rT} \phi(S_T) \vert \mathcal{F}_0 \right] = E^{\mathbb{P}} \left[ e^{-\left(r+\frac{\lambda^2}{2}+\frac{\lambda}{T} W_T^{\mathbb{P}}\right)T} \phi(S_T) \vert \mathcal{F}_0 \right] $$
This shows that, under BS assumptions:
- The option price can be calculated as an expectation under $\mathbb{Q}$ in which case we discount cash flows at the risk-free rate.
- The option price can also be calculated as an expectation under $\mathbb{P}$ but this time we need to discount cash flows based on our risk-aversion, which transpires through the market risk premium $\lambda$ (which depends on $\mu$).
This answers your question:
Therefore, in addition to use expected returns, which other adjustment might be needed in order to estimate real-world probabilities?
You need to use a stochastic discount factor accounting for the risk aversion, see above and further remarks below.
Estimating real-world probabilities assuming BS
You have different possibilities here. The first idea which springs to mind is to calibrate your diffusion model to observed time series. When doing that, you hope to get an estimate for $\mu$ and $\sigma$ in the GBM case. Now given what we just said earlier, you must be very careful when pricing under $\mathbb{P}$: you cannot discount at the risk-free rate. Also obtaining a statistically significant estimation for $\mu$ (and the latent equity risk premium) may not be as easy as it seems see the discussion here
It's more complicated than that when you choose another model than BS
The relationship:
$$ V_0 = E^{\mathbb{Q}} \left[ \frac{V_T}{B_T} \vert \mathcal{F}_0 \right] = E^{\mathbb{P}} \left[ \frac{V_T}{B_T} f \vert \mathcal{F}_0 \right] $$
with
$$f = \left. \frac{d\mathbb{Q}}{d\mathbb{P}} \right\vert_{\mathcal{F}_T}$$
will hold (under mild technical conditions).
Compared to the risk-free discount factor
$$DF (0,T):=1/B_T $$
the quantity
$$SDF (0,T):=f/B_T$$
is best known as a Stochastic Discount Factor (maybe you've already heard about SDF models, this is precisely that) and we can write, without loss of generality:
$$ V_0 = E^{\mathbb{Q}} \left[ DF (0,T) V_T \vert \mathcal{F}_0 \right] = E^{\mathbb{P}} \left[ SDF (0,T) V_T \vert \mathcal{F}_0 \right] $$
The problem is that, depending on the model assumptions you use, you cannot always have a simple and/or unique form for $f$ (hence $SDF (0,T) $) as it used to be the case in BS.
This is notably the case for incomplete models (i.e. models that include jumps and/or stochastic volatility etc.). So now you understand why when we need models to price options, we directly calibrate them under $\mathbb{Q}$ and not on time series observed under $\mathbb{P}$.
