If you have the other side out... can't you use a metal rod to hit on from the outside ? So you are applying pressure exactly in the direction it needs to move..... Maybe it has worn a groove at the bearing which it is laying in enough to keep it from sliding out....