Just to clarify, the cage is a sheet metal or plastic ring that gives the balls proper spacing. The cage can wear out and the balls fall to one side and no longer support the inner race relative to the outer race.

In any event, remove the bearing and have a look. I've seen some bearings in other applications spin the outer race and wear the part the bearing presses into.