Do you have an actual gauge reading of the Vacuum?
It is possible for everything in the Vacuum System to work when it leaks just a bit but for it to cause a cumlative issue with the shut off.
To troubleshoot that disconnect all the vacuum stuff going into the Fire Wall (accept for the the vacuum line going to the vacuum shutoff on the Fuel Injection Pump) and plug off the vacuum side and see the shutoff is normal after that. If so you have a vacuum leak inside of the Car.
If your vacuum is not leaking and your Vacuum from the Vacuum Pump is high enough all that leaves is the Vacuum Shutoff Valve that is on the Steering Colum Lock and on my Vehicle there is little 2/3 way valves on the valve cover that control the EGR. Those 2/3 way valves can leak especially if the little levers are worn off.
See post #23 for some pics of the steering colum shutoff valve.
http://www.peachparts.com/shopforum/diesel-discussion/382612-84-300d-why-cant-i-keep-thing-running-2.html