Caster moves the wheel forward and backward in the wheel well. If you are that far further back on one side you should distinctly pull to the direstion of the side with the tire to the rear.
Caster definitely should be checked but you should have a 4 wheel done by someone that can measure set-back, wheel base difference, and track width difference. Looking especially at the setback and wheel base issues. This will tell you if the car has been "diamonded" (sp?) or just punched if caster doesn't correct the situation.
Bosch Master, ASE Master, L1
33 years MB technician