I think the shaft could sleeved to bring it up to size. There's little radial force on it. Or you could bush the bearing.
What is the number of the pilot bearing?