I did exactly as you described but got it out without incident - but I had a helper.

Removed the trim, cut away the gasket from the outside, then two of us pushed at the top and it came loose.