This is just a guess based on stuff I'd previously read. It sounds like the soldering is bad on the flip side of the button. You'd have to take the panel out in order to solder the connection.

Good luck with it, and again, this is just my guess. Some people even hit their panel to sometimes make that connection work. If it does go in and out based on bumping, it's definitely the connection.

1991 300d