The rubbers were toast...its more noticeable once they are completely removed. They go bad with age, not mileage. Mine were cracked and done @ 70k on the odo, I just didn't get around to doing them until 105k.
It stopped moving around under acceleration on the freeway (Accelerate it drove to the right, foot off throttle it would return to the left) and blowing in the wind. That's my best description of the change.
FWIW, unless you live in the rust belt, your brackets are probably fine. You will see the bushings are probably bulging and that's when you know its time to replace.
EDIT: Here's my thread with photos for the control arms:
This is why W124's blow away in the wind / wander