With careful bolt and bushing maneuvering, I didn't have to remove the exhaust manifold on our 2.3. If yours is the 2.6, that may not be the case. However, it's also possible your motor mounts are worn out, reducing clearance. That caused motor vibration and noise for us, but not shimmy. Still, a loose drivetrain might have unexpected consequences... Just a thought, if the control arm bushings don't solve the problem.

Steve

__________________
'91 MB 190E 2.3
'08 RAV4 Ltd 3.5
'83 Lazy Daze m'home 5.7