First, let me give you some answers based on my intuition on how to hedge that sort of position. Hopefully, they'll help a bit.
When delta hedging a call on a stock, you take the necessary amount of stock in order to neutralize delta. But with caplets this isn't as intuitive, you can't just buy a forward rate, so how do you delta hedge a caplet in the first place?
You can still hedge using forwards in this case. The idea of buying $\Delta$-times the stock for replicating the call is to reproduce changes in price of the option via a sensitivity times the movement of the stock, which is just a first order Taylor expansion.
In the case of a caplet, you can do similar buying an appropriate amount of a forward for that same expiry, note that you don't need to buy the same exact rate, you could just enter an ATM FRA. In that case, your goal is that if rates go higher/lower than the spot, you'll have a P&L from your forward, appropriately adjusted via the sensitivity of the caplet (that you computed before, in order to chose the amount of hedging needed).
Any gain or loss resulting from buying/selling the stock is borrowed/lent at the risk free rate. With caplets, you work under the forward measure, does that mean I have to borrow/lend at the rate underlying the zero coupon bond?
Here, I assume you can still borrow/lend at the same rate. However, I think that entering an ATM FRA, which would cost zero at start, has the same hedging effect, therefore you wouldn't need to borrow money to hedge in that case. This as far as using delta for hedging your position.
Edit
In the case that your goal is to obtain the value of the caplet by using a replicating portfolio the instrument you would have to use are the zero coupon bonds (ZCB). As in BS for equity, you should use basic assets in your replication portfolio, and invest/borrow the remaining cash.
Let's say you want to replicate the value of a caplet of tenor $\Delta T$ that fixes at time $T$ and that you'll be using a time step for readjusting your position of $\Delta t$. In this case you would be buying/selling ZCBs with a payment date equal the tenor of the caplet you'd like to hedge plus the time step used for hedging, that is, at time $n \Delta t$ the delta term in your portfolio and the amount of cash you'll be investing/borrowing would be
$$\Delta_{n} \mathcal{B}(n\Delta t, \Delta T + n\Delta t), \qquad X (n\Delta t) ,$$
at time $t = (n+1) \Delta t$, you have to
- sell the $\Delta_{n}$ ZCBs you bought, now with price $\mathcal{B}((n + 1)\Delta t, \Delta T + n\Delta t)$
- buy $\Delta_{n+1}$ ZCBs expiring at $(n+1)\Delta t + \Delta T$, of price $\mathcal{B}((n + 1)\Delta t, \Delta T + (n + 1)\Delta t)$
- invest/borrow
$$X(n\Delta t) \cdot (1 + r_n \Delta t) + \Delta_{n} \mathcal{B}((n + 1)\Delta t, \Delta T + n\Delta t) - \Delta_{n+1} \mathcal{B}((n + 1)\Delta t, \Delta T + (n + 1)\Delta t)$$
Probably I am not using the clearest notation. I encourage you to read chapter 6.3 of S. Shreve's book "Stochastic Calculus for Finance I: The Binomial Asset Pricing Model". Moreover, you can find some help as well in the solutions to that book problems here.
Maybe it becomes clearer now.
