I'd go ahead and replace it pronto. I had the same symptoms and ignored just a little too long. It suddenly and completely failed while out of town. I had to have the car towed, etc. etc. It was a real headache. The part is only about $25, and installation is straightforward.

If you don't get it replaced, learn how to jump the starter via the solenoid from under the hood. That way you can start the car even if the neutral switch goes out on you.