You talking about the interior or exterior mirror? Interior is a breakaway design. Just pull it out. If it's sticky spray some WD40 in it then pull it out. Not hard. I've done it on a 25 year old car without the help of WD and it just came right out. You're probably doing it wrong.

__________________
'02 BMW 325i
'85 300D 450k
'93 190E 2.6 170k(killed by tree)
'08 Ducati Hypermotard 1100S 6k
'06 Ducati S2R800 14k(sold)