Wow, without the 19mm bolt, the only thing holding the tensioner on is the friction with the little front brace, and the adjustment rod. I would think it would slip around quite a bit.
The turn/click feel you are getting on the adjuster is possibly the end of the rod screwing all the way out of the threads. The rod has lefthand threads, so tightening the bolt CWise drives the rod downward, and eventually right out of the adjusting tube. So, if the rod has screwed out the bottom of the adjustment tube (look down in there and your should be able to see) then there is nothing but the front brace pressing on the tensioner to keep it from just sliding down and out.

Bettter not drive any farther than you want to walk home- when it slips a bit further, the belt will slack, and you will have no coolant pumping.

Oh, just replace the tensioner when you do the fix. In 20 years of owning Old Pearl, the only time she ever let me down was when the rubber tensioner just suddenly snapped in a parking lot 80 miles from home.

DG