The blower speed depends on the differential between the vehicle interior temperature and the setpoint.

If the setpoint is 20°C. and the interior of the vehicle is 30°C, it's going to sit on the #5 blower speed position for quite awhile until the temp. drops to 23°C or so.........then, and only then, will the blower speed begin to slow down.

If the temperature never gets close to 20°C., the blower will never leave the #5 position.