If you have a delta-hedged position and you're short gamma, why are spot price movements bad?

I simply can't wrap my head around the concept of Gamma. I've read multiple sites and explanations and for some reason can't wrap my head around the logic, so I feel that it'll really help for me to explain my understanding and have someone point out where I'm incorrect/flawed. Apologies for the elementary question.

If I have a delta-hedged position (so delta = 0) and I'm short options, this means I'm short (negative) gamma, from what I've read. Now, my thoughts are that if the spot price increases, this implies that my delta is now negative, which means that as the spot price increases, the option price is decreasing. If I've sold this option, why isn't this fall in price good for me?

I know that I'm incorrect, I just don't understand $why$ or where I'm going wrong. Any help or pointing out my errors would be much appreciated. Thank you