Glue wont work.
The part that breaks is the plastic housing that hold the switch in. Poor design. New design cars, the switch breaks and then you buy another switch. Back to the fix. Looking up the console parts on your car, you will find that the plastic part (that is glued to the wood) is available. You have to remove the broken plastic housing part from the wood and replace the plastic. I didn't remove all the plastic, but carefully cut out for 2 switches and replaced. I glued the plastic to the wood using gorilla glue for wood,stone, metal, ceramics ect. I also predrilled for 2 very small wood screws. Turned out real good.
When you have the console out look at the plastic housing for a part number.

__________________
2009 CLS550 55,000
2004 ML350 144,000
2004 X5 95,000
2002 X5 165,000
1996 320S 155,000 (sold)
1991 190E 192,000 (sold)
1989 300SE 160,00 (sold)
1984 300D 210,000 owned since 85, (sold)
1984 300SD 160,000 (sold)