Don't grab the mirror, grab the mount and pull. It has spring loaded balls that hold it in place so that it can easily pop out in the event of an accident.

Hope this helps,