I think you either have a sticking solenoid ( which is just a big electrical relay which also pulls the teeth on the starter forward to engage the flywheel ) or something wrong in your key switch....
If it is the key switch... you might investigate to see if you could just cut that wire to that switch... and reroute it to a normally open , press on switch to use for the starter... old cars had this seperate switch for the starter ( my 54 f500 has it... and all my tractors )... but the nice thing is if you hide the button ( available at most auto parts places ) it becomes an antitheft device.
And that may be much less expensive than fixing or replaceing that switch....