Major irritation, for sure.
I suspect you will have to have someone who really knows what they are doing set the car up "off spec" as Steve suggests, to counteract the pull. Something isn't quite right, and it is very likely a minor frame distortion you aren't going to be able to eliminate. If, for instance, the rear subframe sits just slightly off perpendicular, unless someone knows exactly how to handle this, you will end up with some drift. Ditto for a tiny bit of stagger in the front suspension.
If it hasn't been handled this way, you need to find an alignment shop who knows how -- just setting to factory numbers won't help.
I do know that MB, in the W108 manual, warns that you must stay on the positive side of the camber spec for the driver's front, or it will pull to the left with just the driver in the car.
My 300TE now drives with no pull at all, and is almost totally immune to cross winds -- I couldn't figure out what the funny roaring noise was until the cars around me started moving sideways -- big thunderstorm blowing up and I couldn't tell at all! So it can be done, you just have to find the right guy.
1972 220D ?? miles
1988 300E 200,012
1987 300D Turbo killed 9/25/07, 275,000 miles
1985 Volvo 740 GLE Turobodiesel 218,000
1972 280 SE 4.5 165, 000 - It runs!