Possibly also just drilling a hole or two into the face of the race (difficult because it is hardened material) could create a lip where your drift can get some purchase and allow you to hammer it out.

__________________
Prost!