Caster in not generaly considered to be a wear angle. Especialy a the low degree that you are at. High degree setting ie.10deg and above can cause some wear. Mostly with agressive type cornering. I dont think that this is contributing to your problem.