I have been trying to learn HW1F on my own, out of nothing more than genuine curiosity during my twilight years, and I'm confused on the issue of calibrating. I don't know why, but all the research papers seem to be missing something, or there's some crucial detail I just don't understand regarding the calculus.
I am using Brigo and Mercurio as a guide.
Suppose we are given a the process as
$dr(t) = [v(t) - ar(t)]dt + \sigma dW(t)$
with the market instantaneous forward $f^M(0,T)$ as
$f^M(0,T) = -\frac{\partial lnP^M(0,T)}{\partial T}$
where $P^M(0,t)$ is the market discount factor. Then we are given
$v(t) = \frac{\partial f^M(0,T)}{\partial T} + af^M(0,T) + \frac{\sigma^2}{2a}(1 - e^{-2at})$
where $a$ is an input to the model where practitioners typically use 5% (this point also making me uncomfortable, since 5% seems very arbitrary).
At first I was thinking there was something magical I didn't understand about the partial derivative in $f^M(0,T)$. But then I suppose if $P^M(0,t)$ is observable by the market, or rather, $f^M(0,T)$ is the z-curve, is it not?
From there, I could just fit a cubic spline through a few points on the curve to get the $f^M(0,T)$ for any $T$.
And here is where my 40 years post-calculus begins to show... is there some simple trick for understanding the partial derivative $\frac{ \partial f^M(0,T) }{\partial T}$? I believe $-\frac{\partial lnP^M(0,T)}{\partial T}$ is just $\frac{-ln(P(0,T))}{\Delta T}$ from my understanding of continuous compounding, but I have long since forgotten why.
Then I supposed, once I calculated a few $v(t)$ from the observable market, then I could fit a spline the $v(t)$ to get a $v(t)$ for any $t$ within the range of observed $f^M(0,T)$. Does this sound reasonable?
Coming to the $\sigma$, I have seen several comments alluding to calibrating the volatility to a set of swaptions or caplets, depending on the types of things one is trying to model. Why would you not just take a treasury option Black-Scholes implied vol for calibrating $\sigma$? I guess this is because the rate being modeled is the instantaneous short rate and not the yield-to-maturity on the zero-coupon bond associated with $P^M(0,T)$? Beyond this, I get completely lost. I have seen some commentary on using the diagonal of the volatility surface for calibration, but I'm not sure why.
So my question(s) then is... is the approach for the $v(t)$ correct? Should I question further where $a=$ 5% comes from, or take it as a given? Why is the diagonal of the swaption surface used in calibrating $\sigma$? Lastly, I notice that Hull and White advocate a lattice for calibration. But then some approaches I believe calibration is done without a lattice using possibly a root-solver. Is this my ignorance? Does (mostly) everyone use a lattice to calibrate?