Leaking delivery valve holders can potentially cause idle issues, though the O-ring part is not on the high pressure "delivery" side of things.

How old are your injector nozzles? If the pop pressures are off or if you have a poor spray pattern, you can expect a bit of roughness, especially at idle.

Compression? A cylinder imbalance of >25PSI can contribute to rough/lopey idle. The bigger the imbalance, the more roughness you can expect.

Don't forget that it's an IDI diesel, it's just gonna do what it's gonna do. A perfectly smooth idle probably isn't a realistic expectation. Believe it or not, the fuel you're running can affect the idle quality.

__________________
Current stable:
1995 E320 149K (Nancy)
1983 500SL 120K (SLoL)

Black Sheep:
1985 524TD 167K (TotalDumpster™)

Gone but not forgotten:
1986 300SDL (RIP)
1991 350SD
1991 560SEL
1990 560SEL
1986 500SEL Euro (Rusted to nothing at 47K!)