Nobody is saying that swaps are "riskless". The question is whether the "appropriate" interest rates should be "riskless" in the first place.
The assumption they should is a convention borne of decades of academic papers and economic textbooks. For any academic, investors are always sitting on a dollar of cash and working out how much to allocate to a risky asset versus a T-Bill. Except even they will probably accept that in reality that the cash probably sits in a bank CD or Apple commercial paper etc. if not put work into riskier (as opposed to risky) assets.
The case for using swaps is about an appropriate baseline, against which to take stock risk. If I buy an S&P future, my return is an excess return relative to my cost of carry, which is a function of private-sector interest rates not public sector ones. Stock + Dividend = Cash + Future.
This might be equal to Bond Yield + Carry&Roll + TED spread... but why should the TED spread or 1Y5Y rolldown make the S&P any more or less attractive? Plus if Uncle Sam can finance his balance sheet more cheaply than you or I, then what good is that to us? We should compare stocks to the funding costs at which market participants are funding their balance sheets. These are the interbank rates baked into futures pricing.
If we wanted to think about equities 1 or 5 years out, then the swap rate is the carry cost of the Cash above. It's the genuine opportunity cost/benefit of not taking equity risk. Academics are academics; and market participants are market participants. The two groups just use slightly different conventions here.