Quite possibly worn brushes on the blower motor. When they get short enough, they can jam sideways in their housing and the springs cannot push them against the armature (is that the right part name?). Pull the motor, visually inspect it, and test with a straight 12v.

Suppose it could be the transistorized speed controller, but I've not heard of those going intermittent before failing.

- JimY