It looks difficult because there is not an intuitive sign on how to do it. But the mirror is designed to break away in an accident, so there are some heavy duty springs in it that clamps it to the car. Just pushing on it will disengage it from where it seats. In my case I guess it just worked itself loose over time. I applied some thread lock and screwed the seat back in. To reinstall it you can probably figure it out from looking at the mirror.