I'm a little confused...are you talking about the plastic ball joint that is behind the mirror assembly? The mirrors are pretty easy to swap as assemblies (from where it mounts to the car roof). You just have to yank pretty hard and it'll come off. My rear view mirror was cracked badly and it took all of 2 minutes to swap. It's a little bit of a pain to get the replacement mirror in but that's it.

I don't think you can get over the issue that there's quite a bit of material overlap if you're considering removing material...I'd feel it'd be too much material from the ball and something might crack if you push too hard. Maybe you'll have some luck with heating the plastic.

__________________
1982 240D, sold 9/17/2008
1987 300D TurboW124.133 - 603.960, 722.317 - Smoke Silver Metallic / Medium Red (702/177), acquired 8/15/2009
262,715 and counting