As sayed by Matt, It is most likely your Vaccum pump.One of the valves inside is stiking. Possible a main leack,but if the brakes worked fine before and nothing was done under the hood to brake the main black hose from pump to booster than i think it is the pump itself.The is a repair kit available to rebuilt the pump,or just replace it