Sticky starter ONLY when cold outside

On cold days, I turn the key to the start position and the starter clunks (Like it's connecting with the flywheel) then it runs free. Repeating this cycle two or three times will usually get it to connect and start the engine. I've never hit the starter and it never does this when the air above 40*f. If the engine is warm or it's a warm day, it works flawlessly.

I had this problem last winter and it went away completely once it the days got warmer. Now that it got down to 35* today, it's happening again.

Any ideas on what it's doing or how to fix it? Is the starter about to cr@p out?