This is a possibility but just leave the fuel cap off or remove it temporarily when you do not expect it to start. Do this as soon as you shut down in case suction is pulling your fuel back as the car sits making the restart hard.

If this were the case the lift pump valves may be leaky as well. They act as the return check valves in the fuel system as well as enabling the lift pump to function.

In reality there are many possibilities for your problem.