Assuming that the vacuum system itself is okay, there are three major possibilities:
Failed check valve in the booster line.
Poor seal at the master cylinder/booster interface.
Leaking booster, which will require replacement.