Take a liquid market like US T-bills, notes and bonds... Who would want a long-term security that yielded less than the short?
Even if someone is a long-term lender, why wouldn't they hold the short-term, higher-yielding securities then switch to the long-term securities if and only if they yielded more? Why wouldn't such arbitrage prevent the possibility of an inverted yield curve?
There are many reasons why a yield curve can be inverted. A default-free yield curve reflects a combination of -
Let's consider a case where market expectation is flat (people don't think interest rate will rise or fall in the next 10 years) and bond risk premium is zero. You may think the yield should be flat, but it actually would still be inverted. This is because longer term bonds have higher convexity, so investors are willing to accept a lower yield in order to gain this "convexity advantage."
Now consider a world where convexity isn't considered valuable and market expectation is still flat, but bond risk premium can be negative, causing the yield curve to invert. Throughout much of 2012 and 2013, US bond risk premium was indeed negative, partially because of QE, and partially because of flight-to-quality flows (Europe was blowing up!)
Finally, the yield curve can be downward sloping because the market expects interest rate to decline in the future.
I'm not sure why you think there's an arbitrage when the curve is inverted. Consider a scenario where 1-year rate is 5% and 10-year rate is 4%. An investor then purchases the 1-year note purely because of yield considerations. But there's no reason why the yield curve can't flatten more. Let's say 1-year rate rises to 6% (maybe because the Fed keeps hiking short-term policy rates), and 10-year rate declines to 3.5% (perhaps because of Foreign central bank buying). In this case, the 1-year bond LOSES money, while the 10-year bond actually makes money.
Inverted curves (typically) appear when the economy is overheating. There is full employment but investment demand is still there and it is creating inflationary pressures. The central bank increases the short rate (which is their classical policy instrument) to take money off the table and cool down investment demand. However, the market knows that this is part of the cycle, and expects rates to go down in the medium term as the economy cools down.
In this scenario, holding the short dated security might not be optimal as this requires purchasing another security when the first matures, at rates which will most probably be lower and might be highly uncertain. For example, the central bank can tip the economy into a recession and then cut the rates dramatically in response: then one would face very low yields when rolling the position over.
The market will price all these eventualities and produce an inverted curve. There is no issue of arbitrage or irrational behaviour.
Another instance of inverted curves occurs when a sovereign is in trouble, but this is more related to credit risk rather than macro conditions.
