The bank in which you deposit your capital doesn't create the money to pay your interest; it earns it by lending to someone who can make good use of the money. The textbook example is a farmer who borrows £1000 to spend on fertiliser, gains £1200 in extra crop, and repays the bank £1000 plus £100 in interest. The bank gives you £50 for the use of your money, and all three of you are happy. But notice it doesn't actually matter to you whether the bank lends the money to a farmer, a startup company, or somebody with a foolproof system for beating the Las Vegas casinos; the bank pays you an agreed sum for the use of your money, no different in principle from paying rent for an office. If the bank makes better use of the money than you would (as they should do, being professionals), they make a profit from helping someone else create real wealth. That's perfectly logical and sustainable, indeed at best it's self-reinforcing; next year the farmer borrows £5000 to buy and plant a new field once the bank works out that his gains will more than pay for the interest; hey presto, your money has helped produce food that would not otherwise exist, and made a small profit for you while doing so.
It gets more complicated when the bank makes loans (directly or through a central bank) to the government. But though a government can just print more money when it has to repay the loan (indeed that's the only way it can work; how would you expect the US Treasury to redeem a dollar bill except by giving you another dollar?) they are not exempt from the laws of economics. Those governments that run a deficit are borrowing money; from bond purchasers, from foreign banks or in the last analysis from anyone who accepts a banknote in payment of a debt. Most governments have a lot of security to pledge, so are considered good credit risks. And the things they spend the money on are, in theory, large-scale versions of buying fertiliser to improve future crops; the British National Health Service was justified by the Treasury of the 1940s on the grounds that reducing the number of days that workers were off sick would more than pay for the hospitals, and the increased happiness would be a bonus. (Whether that actually worked depends on what accounting you use).
So governments, just like anybody else, borrow money in the belief that the good they can do with it justifies the amount they will have to pay for the use of the money; they can continue to do this as long they invest wisely enough, and pay interest rates high enough, to maintain confidence in the currency. In the 19th century, banks that issued their own currency sometimes issued more than they could redeem, had speculators buy up the notes and present them for redemption all at once, and went bankrupt. Your government could in theory do the same, in which case the money you keep in that currency (as opposed to in usable property) would be worthless, unless it has the artistic value that Confederate or Czarist banknotes have now. Most of us think the risk low enough that it's worth rising above a barter economy.