Same thing happened to me and it turned out to be the blower motor. I opened mine up and the brushes were worn down so there were barely making contact. I found some brushes thar were close to fitting and modified them to work. That was 3 monthes ago and since then I purchased a new blower motor with the thought of replacing when weather turns warmer. You can buy a neew motor without blower for under $100. and then just swap out the blower.
Hope this helps, this had me baffled for a while.
My car is a 83 380sec w 164K. It seems that this is common with older cars.