it could be the brushes, as the way speeds are adjusted is the voltage is lowered. low voltage may not cross the gap of bad brushes as easily as the high speed.
that said, it could also be the "hair dryer" box that reduces the voltage. or of course, the climate control...