"I will measure the voltage of the engine when it's off and running and will adjust the belt tensioner based on the reading. However, what'd be the correct voltage value range for these repective cases?"

Don't use battery voltage (or any other voltage) as a gauge for adjusting belt tension. Since you had the belt off, I assume you checked its condition and it's ok. In that case, the gauge/pointer built-in to the idler assembly attachment is the one to use.

If the problem persists after verifying the tension, you likely have a problem with the alternator. Output voltage should exceed 13.5V under heavy load, >14V with light load.

Steve