So, from this simple no-arbitrage argument, we see that the price of the option must always be at least its intrisic value.
Yes indeed
However, at this point I realized something strange: if this is true, why in the world should I exercise my put option before expiry?? The inequality seems to indicate that it would be an unwise decision to ever exercise the American put option at time $t$, and so, the only right time to exercise an American put option would be at expiry
How does the fact that $P(t,S_t;K,T-t) \geq (K-S_t)^+$ gets you to that conclusion? This is a completely fallacious reasoning IMHO.
The price of an American option is:
$$ P(t,S_t;K,T-t) = \text{sup}_{\tau \in [t,T]} \mathbb{E}_t^\mathbb{Q}\left[ e^{-r(\tau-t)} (K-S_\tau)^+ \right] $$
the inequality you observe can be obtained by splitting up the family of stopping times $\tau$ with values in $[t,T]$ using the fact that
$$ [t,T] = \{t\}\ \cup\ ]t,T]$$
We then get,
\begin{align}
P(t,S_t;K,T-t) &= \text{sup}_{\tau \in [t,T]} \mathbb{E}_t^\mathbb{Q}\left[ e^{-r(\tau-t)} (K-S_\tau)^+ \right] \\
&= \max\left( \underbrace{\text{sup}_{\tau=t} \mathbb{E}_t^\mathbb{Q}\left[ e^{-r(\tau-t)} (K-S_\tau)^+ \right]}_{\text{immediate exercise}}, \underbrace{\sup_{\tau \in ]t,T]} \mathbb{E}_t^\mathbb{Q}\left[ e^{-r(\tau-t)} (K-S_\tau)^+ \right]}_{\text{differed exercise}} \right)\\
&= \max\left( (K-S_t)^+, \sup_{\tau \in ]t,T]} \mathbb{E}_t^\mathbb{Q}\left[ e^{-r(\tau-t)} (K-S_\tau)^+ \right] \right) \\
&= (K-S_t)^+ + \max\left(0, \sup_{\tau \in ]t,T]} \mathbb{E}_t^\mathbb{Q}\left[ e^{-r(\tau-t)} (K-S_\tau)^+ \right] - (K-S_t)^+ \right) \\
&\geq (K-S_t)^+
\end{align}
When the holder has to choose whether or not to exercise at time $t$, he/she should compare the value of his option position $P(t,S_t;K,T-t)$ with the payoff he/she would get if he/she exercised immediately (intrinsic value) $(K-S_t)^+$.
$$ \underbrace{(K-S_t)^+}_{\text{immediate exercise}} - \underbrace{P(t,S_t;K,T-t)}_{\text{option value}} = \max\left( 0, (K-S_t)^+ - \sup_{\tau \in ]t,T]} \mathbb{E}_t^\mathbb{Q}\left[ e^{-r(\tau-t)} (K-S_\tau)^+ \right] \right) $$
The holder would then exercise at $t$ if the RHS is positive, that is iff
$$ (K-S_t)^+ \geq \sup_{\tau \in ]t,T]} \mathbb{E}_t^\mathbb{Q}\left[ e^{-r(\tau-t)} (K-S_\tau)^+ \right] $$
or equivalently if the intrinsic value is greater than the continuation value. You see that it is not possible to make a general claim such as "the only right time to exercise would be at expiry" from the above.
[Edit]
Let $$\frac{dS_t}{S_t} = rdt + \sigma dW_t^\mathbb{Q} $$
Elaborating on @MJ73550's remark if $(S_t)_{t\geq 0}$ is a martingale, that is if $r = 0$, one can show that, for any convex function $\phi$,
$$ \mathbb{E}_t\left[ \phi(S_\tau) \right] \leq \mathbb{E}_t\left[ \phi(S_T) \right], \ \ \forall \tau: t \leq \tau \leq T $$
to see this, we can appeal to Jensen's inequality along with the optional stopping theorem. Indeed for all stopping time $\tau \leq T$ we can write:
\begin{align}
\mathbb{E}_t\left[ \phi(S_T) \right] &= \mathbb{E}_t\left[ \mathbb{E}\left[ \phi(S_T) \vert \mathcal{F}_\tau \right] \right]\ \ \text{(Tower property)}\\
&\geq \mathbb{E}_t\left[ \phi(\mathbb{E}[S_T \vert \mathcal{F}_\tau]) \right] \ \ \text{(Jensen's inequality)} \\
&= \mathbb{E}_t\left[ \phi(S_\tau) \right] \ \ \ \ \ \ \ \ \ \ \ \ \ \text{(Optional sampling theorem)}
\end{align}
Using this result the price of an American option, when $S_t$ is a martingale becomes:
$$ V^{AME}(t,S_t;K,T-t) = \sup_{\tau \in [t,T]} \mathbb{E}_t^\mathbb{Q}\left[ \phi(S_\tau) \right] = \mathbb{E}_t^\mathbb{Q}\left[\phi(S_T)\right] = V^{EUR}(t,S_t;K,T-t)$$
Now, if we consider the case $r \leq 0$, the stock process $(S_t)_{t \geq 0}$ becomes a sub-martingale since in that case:
$$ \mathbb{E}_t[S_T] = S_t \underbrace{e^{r(T-t)}}_{\leq 1} \leq S_t $$
Should we let $\phi: x \rightarrow e^{-r(\tau-t)}(K-x)^+$ which is still a convex function of $x$, we would see that for put options, where $\phi$ is a monotonically decreasing function of $x$, applying a similar reasoning as earlier shows that it is never optimal to exercise the American put before maturity $T$.