RPM Control is the function of the regulator, so if you have constant supply voltage to both the reg and blower, then you have an intermittant problem , possible in the regulator b/c running voltage is higher with charging system.
Or, you have a bad ground at regulator,.

The test will be to jumper the blu wire to ground at the blower..this bypasses the regulator and gives fan full power [ high speed].
If the motor still slows down with constant voltage , you have a blower problem..if not, then I suspect regulator