Seemingly random loss of power

[Yumeko-chan]

...except that most of the time it happens when I have to stop by my bank!

Anyway, every now and then I'll stop by my bank, run in and use the ATM or whatnot, come back out to start the car, and suddenly it won't wanna start. It'll crank and crank, and when it does start it's like the car has very little power at all, like it'll die at any moment. When that happens I can turn the car off, wait a few seconds, and try it again. It may take a few tries before it'll start and run like normal. And like I said, it mostly happens when I go to the bank! =P

(If I try to drive when the car is lugging like that, it'll actually take a few seconds after I step on the gas pedal to start moving, and slowly at that, but once it gets up to speed it seems fine, at least until I get to a red light or a stop sign)

What could be the problem here? It doesn't happen often, but enough that it annoys me when it does.