The tension adjusting nut is reverse threaded and really soft metal. If you extend it past its throw you will strip it. Checking the pulley on the tensioner for play is a good suggestion. As far a shimming the bushing goes, the rubber gromet that insulates the bushing preventing metal to metal contact does so both laterally and vertically. Even if lateral play is nil, you might be getting the rattle from vertical movement which would be tough to detect by moving the piece manually. Aside from using a mechanic's stethascope, a short piece of broom handle or dowel pressed to the bone behind you ear and then touched to various parts of your engine will help narrow down rattles.

Order the adjusting bolt and nut while you're at it. It is cheap and I'll bet you will need it if you've been trying to adjust it without success. Also make sure you get the right size belt. The difference between the two that I show for the 104 engine (2440 and 2445) is tiny but makes all the difference.

Good luck