From what I've read it sounds like your blower motor brushes are worn and/or making poor contact. Someone on ebay is selling a set of brushes at an opening price of $26. I don't know where else to get brushes. A new motor is a little over $100 from FastLane.
Check the axial play of the blower motor when you have it out. As the bearings wear, you'll hear the fan cage scrape against the motor housing. And then it really gets fun.