Since you have mentioned taking first differences as a possible remedy for nonstationarity and since this remedy is mainly appropriate when dealing with I(1) processes, let me focus on whether these time series are I(1) or perhaps I(0).
(I(1) implies nonstationarity while I(0) does not imply it but permits it.)
If YoY % change in CPI and the nominal Treasury yield were I(1), i.e. contained unit roots, they could wander off to $+/-\infty$ and never return. This is clearly not the case as e.g. you have strong economic arguments against a scenario where the nominal Treasury yield is negative and large, and you would not think inflation or deflation can grow without bound and never come back. Thus, these processes do not contain unit roots.
What about modelling these processes as if they contained unit roots? If you work with relatively high frequency data on the processes, you may find rather strong persistence, suggesting that you can approximate the processes reasonably well using unit-root models. They would work fine over not-too-long time horizons. If you work with low frequency data, the persistence is less strong, and unit-root models do not approximate their behavior that well. A stationary model may then be more useful. Hence, what model you choose could depend on the frequency and time span of the time series.