I had the same symptoms with a Jag XJS that ended up being a bad chassis ground. It had the battery in the trunk and the battery ground strap connection to the chassis had some corrosion/rust. Once this was cleaned up I never had this problem again.

I thought that it was a sticky solenoid because tapping the back of the starter with the solenoid energized got it started. Thats how the marks on the back of your starter got there.

As far as the starter commutator having a bad spot from HarryM. I am wondering how the starter can advance past the bad spot.

Bad / sticky solenoid maybe because the starter motor does not get power untill the solenoid engages. If you pull out the starter and jumper it and it spins ( be careful ) it is more likely a poor power conection. Starters take a lot of AMPS.