It can depend on how big the leak is in the diaphragm. If you have a small tear in the center vent pod, for example, it could do just as you describe.
The default setting for the whole system is defrost vents open. You must have vacuum to close them.
W124 climate control vacuum pod replacement