This problem has been covered before. You can do a search in the archives for mirror, rear view mirror, etc.
The jist is this. There are screws that secure the breakaway mounting to the car that can loosen up. Also, the mirror itself can loosen on ball socket at the end of the stalk after time, and the flipper that changes it for night driving can, but seldon does wear out.
Give it a good hard, but careful, pull and remove it. Check the screws to see if the mounting is good and tight, and see if the problem is in the mirror assembly itself. A real inexpensive fix can sometimes be an application of RTV silicone adhesive to the ball socket, or flipper to help dampen the vibration. But try the mounting screws first.
And, come on Q, you're a big strong man. Replace that mirror!