Have you blocked off and disconnected everything from the pump (including the booster) to test the vacuum directly at the supply fitting on the pump? Sorry if you've answered this already it isn't clear. If you're leaving the booster attached and you have a leak in the diaphragm, low vacuum in the system is exactly the symptom you'd be getting.

The brakes should be holding vacuum even if the rest of the system doesn't. There's a check valve in the brake line to have it hold vacuum. It should have enough vacuum stored to last a pump or two of the pedal before going hard.