Were we doing physics and we said there was an arithmetic Brownian motion we could indeed have a drift rate other than $\mu=r$ and it would make sense. Suppose, for example, that a fluid is moving at velocity $v$ and we have a random walk of particle in it. This would be reality.
The whole point of using SDEs in finance is to identify what ought to be true in equilibrium. Where people go wrong in a lot of interpretation is saying something like this: "if the risk free rate is 5% for no risk, and I demand 10% return to take a risk of $\sigma$, then my drift, $\mu$ ought to be 10%..."
Put another way, we often say that such a person demands a 10% return for a particular risk ($\sigma$). Nature may demand a drift in a moving fluid, but saying an investor demands something of a stock is a bit twisted.
The whole idea of the solution to the SDE for asset pricing is to separate a proposed price process into a drift that is there with certainty and a random element. The first is the drift, $\mu$, and the only drift that is certain is the risk-free rate, $r$.
A better way to say what an investor thinks is to say that, in equilibrium, a stock is priced such that investors have found a price at which there is no selling or buying moving the price, and at which they (collectively) feel the stock is priced such that the expected return is appropriate for the expected risk.
Deep down this is what the risk-neutral measure really means. Under the risk-neutral measure, investors must believe that there is a 50% chance the stock will, in reality, deliver a return that adequately compensates for the actual risk, and a 50% chance of the opposite. More accurately, they are indifferent between holding the bond and the stock at equilibrium pricing taking into account their own estimates of the distribution payoff and their personal risk appetite. In reality this means collectively that the expected return is greater than $r$, though we don't know by how much or what the expected $\sigma$ is either. Due to risk aversion the actual expected return must be above $r$, but we work in risk-neutral space.
Yet one more vector on this is to say that, at equilibrium, an investor with 105 dollars arriving in one year who thinks the stock is fairly priced can either borrow 100 (at 5%) and buy the stock now with the debt paid off in one year, or enter into a forward contract to buy the stock at 105 in one year. These have identical risk and return outcomes. Such an investor believes there will be a return of over 5% this year (or else he would not do the trade) and in fact expects enough return to compensate for the risk. In other words, at the current price, he is risk-neutral (or perhaps a better term is risk-indifferent). Now, if he 'demands' a 20% return of the stock to feel this way he will be sorely disappointed if the stock only delivers a 2% return (or an 11% return or whatever). But, deep in his brain, he expects the stock to go up to (perhaps) 120.
Note that if he is happy with the discount that compensates for risk then he is risk-neutral at that point. Then, of course, it makes perfect sense that the drift is only the risk-free rate since this is the cost of funding. Our investor wants to take on the expected risk for the expected return premium, and the drift, as it were, has nothing to do with risk - only the cost of financing.
Many people find it easier to think of the bond as having a future (riskless) target price of 105, and then the stock to also have a 'target price' of 105 but to be priced, today, at a 'discount' to the bond at 80 (or so).
But, if a bond has a price trajectory of $e^rt$, then a stock must have a price trajectory, in the risk-neutral way, that has a future value well below 105 - something like 85 - so that, when we discount back to today using $r$, its spot price is something like the 80 it must be. And that is where the $-\sigma^2/2$ comes in...
There are some subtleties around what $E[e^{\mu X}]$ is but the real meat of the matter is the above. If we stay in risk-neutral space, which assumes there is adequate expected compensation for anticipated risk, then risk-neutral investors - who would hold the stock at a current price $S_0$ - need a (risk-neutral) price process that delivers a future value of about $S_0e^{rt}$. To get enough 'discount' so that the current price is around 80, in my example, we need to subtract something from $r$ and that something is $-\sigma^2/2$. Try it out on a spreadsheet.