The blower motor is easily accessible in the passenger footwell with the cover removed. Have you actually pulled the fuse out and looked at the ends? Better yet replace it with a new one. I had an intermittent blower on my 84 SD and the fuse looked fine, when I pulled it out there was a hairline crack right near the end cap. These fuses filaments expand and contract do to heat and can fail from metal fatigue without being obviously blown. As the blower motor brushes wear function will become intermittent before failure. If your good with a soldering Iron you can replace them. The only other tip I can offer is to move the ignition key slightly when experiencing your problem as you may have dirty contacts in the switch.

Bob D.