Well, where do we start? I have an '80 300TD which I have basically replaced every part of the ACC system. These two cars are very different though. If I were a wagerin' man I would guess that your problem is vacuum. You are producing vac, but it is taking time to build up enough vauum to close the switch that turns on the blower. Do all your door locks work....all the time?
This is topic that is difficult to attack online. You may want to run a vacuum check on all systems (I think there is 6 or 7). A hand vac pump and manual will do the trick.
Sounds like vacuum to me or a diaphram in one of the many vacuum operated switches located behind the push buttons in the center console. Good luck, but at least you came the right place. These guys know their stuff!