I have only traded a few FX swaps and not in a while so do correct me if my understanding of their technical product specification is not correct:
A Purchase of an FX Swap with say, a 3M tenor, at an initially struck spot rate and forward rate demands the spot receipt of a nominal amount of domestic currency and the payment of rate equivalent foreign currency. At the end of the 3M tenor the same domestic notional amount is paid back and the foreign currency is received adjusted by the forward rate.
The key to your question, I believe, is the fact that the same domestic notional is received back after 3months, when under traditional (no arbitrage) scenarios that amount would have grown by an amount equivalent to the (cross-currency adjusted) rate of interest.
Suppose for EURUSD; spot: 2.0000, 3m fwd: 2.0100, FXFWD: +100 pips.
Suppose EUR interest rate is 4% p/a and in USD it is 6.04% p/a.
Suppose you purchase EUR1000 EURUSD FX swap at 100pips struck at 2.000
Your cashflows look like this from the FX swap:
Date EUR USD
Today 1000 -2000
3m fwd -1000 +2010.1
But don't forget about the interest over 3M:
Intst 10 -30.2
Total 10 -20.1
If the FX rate at the end of the period is 2.01 as predicted then 10 eur == 20.1 usd so you have zero PnL, but if it deviates you have either a small loss or a small gain.
It is this component that you have FX exposure to. The smaller interest rates are the smaller this discrepancy so the risk is reduced.