The tension mechanism is ratcheting and should not move back in once it has pushed out to take up slack. It may be worth your time to remove the tensioner, clean it up, and replace the spring. I did that as part of some other preventive maintenance since the spring is only a few dollars and an improperly functioning tensioner can lead to major problems.