This happens @ normal operating temp? (or close to it?) then its not the glow plugs.

Heat soak is the issue, a connection that is adequate @ room temperature is not gonna handle the amps @ higher temps. I agree w/ T.W. Check/clean your grounds/connections. If everything checks out I'd take a hard look @ the solonoid, followed by load testing the battery, then finally the starter.

__________________
Cheers,
Robert