If the movement at the end of the idler arm is say 2-3 mm it isn't a problem. The older versions that had grease fittings etc. were subject to a lot of wear and allowed the tow in to change with driving conditions. The idler arm should be at the same hight as the pitman arm. That will have a great effect on the way a car steers over an uneven surface. I think it might also effect akerman angles and turning track. The bushings would need to be extremely loose to cause the uncontrolled steering motion described.
Bill