Let us suppose for concreteness that the 10y swap rate is 0.5% today and was 7% a year and and 6.5% a "year minus a day" ago...
reprice the swaps for each historical scenario and calculate returns as the difference between the swaps PV from each scenario and the today's PV
This won't help you at all, really. Take a step back and consider your goal. You're looking at your book's market risk. Should you be concerned about the possibility that the market will jump from where is is today to exactly where it was a year ago? No, that's not a realistic concern and not one you should be worried about. Calculating mark to market (mtm) of your current positions directly using some historical market data will not tell you anything useful here.
Rather, you want to see how the markets moved (changed from one day to another day) historically, and calculate the P&L as the change from the mtm today to mtm if the markets move the same way starting from where they are today.
But what does "the same way" mean exactly? We see that historically, some interest rate moved from 7% to 6.5%; what would "the same" move mean today? I've seen a surprising number of people treating interest rates as equity prices - this is a (6.5-7)/7 = 7.41 decrease, so we likewise decrease today's .5% by 7.41 to get 0.464%. Others look in the change in rate - it decreased by 0.5%, so the same decrease today would be from .5% to 0. Others seek to replicate the percentage change in discount factors (i.e. in the prices of zero-coupon bonds with the same time left to maturity). Still others mix in logs in various creative ways.
(You could be looking at the change in log(1+rate) and applying the same change to today's rate.)
It doesn't help that some regulatory guidance (misguided, in my humble opinion) discuss interest rate stress ecenarios in basis points, for example https://www.fdic.gov/news/financial-institution-letters/2012/fil12002.html says:
Management should ensure it stress tests IRR exposures using
appropriate scenarios, including meaningful interest rate shocks, to
identify the inherent risk. For example, in a low-rate environment,
institutions should run interest rate shocks of +300 and +400 basis
points. If conditions warrant, institutions should test more severe
Once you have the perturbation, you can fully reprice your swaps under the perturbed market data (this is the most accurate if you have enough computing power) to get the perturbed mtm, and subtract the current mtm to get the P&L; or you can escimate the P&L's faster, but less accurately, from the perturbations and the risk measures.