I'd exactly the same symtoms as you do, and after replacing the idler arm repair kit and steering damper, the groan is still there. My power steering fluid is level and I am baffled. I'd lived with it for 6 months since my mech could not locate the fault.
Do let us know if you solved the problem.
1979 W123 (200/M) Midnight Blue
1983 W126 (280SEL) Dupont Deep Black (Previously Pristine Silver)