I would like to make an analogy with equities. To price simple contacts like equity forwards or futures, you don't need a model, you use simple no arbitrage arguments. To price options and other more complex derivatives with non linear payoff you need a stochastic model. It is worth mentioning that when you use a stochastic model to price simple contract like forwards, you get the same price as with simple no arbitrage arguments. Therefore with such a model you can price virtually anything.
For interest rate derivatives, it's the same story. You only need the curve to price simple contracts like FRAs and SWAPs, but to price more complex derivatives like caps, floors, swaptions etc. you need a stochastic model like Hull-White. With stochastic model you can of course price SWAPs or FRAs, but no one does it because it is pointless.
Notice that HJM is not a short-rate model as it captures the dynamic of forwards rates and this is instantenous forward rate model.