Can one use the GARCH model to estimate the realized variance/volatility, such as done in this paper, rather than forecast the volatility, from (high frequency) price/tick data?
You can use GARCH to estimate volatility. If you have high-frequency data, you can use realized volatility to estimate volatility. You would normally not use GARCH to estimate realized volatility (why estimate an estimate?), though I suppose it is technically possible to view your fitted volatilities from a GARCH model as estimates of realized volatilities.