I couldn't agree with you more. I drive my '97 E320 all week long and it is a dream compared to w124. (and I don't have the V-6).
On weekends I drive my '86 300E. There is no comparison. The E320 wins in just about every category. But it's also 11 years newer. The 300E does feel more nimble. Maybe because it's so much smaller.
Good luck - you made the right choice!