Does anyone know how to get the plastic cover of the stem on the rearview mirror?

The mirror is loose between the base and whatever connects it to the window. I am afraid of braking the plastic cover. What's the trick?

Thanks for your help!