As far as I know, a new fuel pump comes with a new check valve. If you changed your pump, then you have a new valve. Your problem is related to what is called " Rest Pressure" or the ability of the fuel system to hold fuel pressure after engine shut off. Two things to look at: 1. Fuel accumulator and fuel system pressure which is bolted to the fuel distributor in older CIS fuel system. Try this link to better understand the CIS fuel system .
http://www.auto-solve.com/mech_inj.htm