This is quite a common problem on 123's. Some people have solved it by cleaning the connections inside the ignition switch. There is a thread here somewhere on how to do it. Other people have rewired the system, putting a solenoid near the battery and only using the ignition switch to activate that solenoid thinking that the old wires and connections in the original system are reducing power to the starter. There's a picture of this alternative in a thread somewhere.
My 85 300TD has this problem periodically. I just carry a piece of wire to jump the terminals in the block after glowing. It usually has the problem a few times in a day or two, then it goes away for a number of months. This has been happening for about 5 years. I originally bought a new starter, thinking that was the problem, but now I just view it as a quirk.
If it got to be a permanent problem, I'd consider wiring a remote start switch directly to the two terminals on the junction block in front of the battery and running the switch up near the driver.

__________________
1977 300d 70k--sold 08
1985 300TD 185k+
1984 307d 126k--sold 8/03
1985 409d 65k--sold 06
1984 300SD 315k--daughter's car
1979 300SD 122k--sold 2/11
1999 Fuso FG Expedition Camper
1993 GMC Sierra 6.5 TD 4x4
1982 Bluebird Wanderlodge CAT 3208--Sold 2/13