I want to replace my fader switch and not destroy the wood panel. How does this come off?

You don't say what you have, but it's probably older and these are all similar. You need to lift up the console wood piece to get at the back of the switch. Then, squeeze the clips together and push it out from the back.
Most console wood facias (around the shifter anyway) come out by removing the ashtray, then the little compartment/tray, then pull wood forward and up (if memory serves, I haven't done it in quite awhile, but it only takes 10 minutes).

__________________
1986 560SL
2002 Toyota Camry
1993 Lexus