Larry is right. Most of time is with the switch, "dirty" contacts. You could do what Larry suggested, pry the switch and clean the contacts, but before you do that, try spraying some WD-40 on the switch.
I had similar problem with my seat switch. Pried open to clean the contact, and it worked for some times, but the problem returned again. Read some threads on this forum, and someone suggested WD-40. Tried the WD-40 trick, and it works so far so good.