Does it go away after the first 3 minutes or so of start-up? The smog pump is designed to pump air into the exhaust to aide in burning of excess hydrocarbons on cold startup. Nothing else. Once the engine is warmed, the EGR system does nothing and the smog pump should be free-wheeling. These pumps are notorious for failing in all sorts of crazy ways. Wide range of neat noises if the internal clutch fails, outer pulley bearings go bad, etc etc. They can also fully seize and shred the belt which is also good times.
I would advise sourcing a replacement sooner rather than later. Even if it's a used pump. Don't pay more than $100 though. Used air pumps at the Pick'n'Pull yards over here are $12.99. I bought the replacement for my car on eBay for $99, fully rebuilt. Got lucky on a bidding auction.
.
You can also remove the smog pump, block off the EGR tube, vac lines. Then get a shorter fan belt for an early car. Although I also believe the fan bearing bracket has to be changed over to the early PN, but not sure. It won't really affect your ability to pass emissions (cold startup ONLY) unless they are really knowledgeable on the visual inspection.
Good luck.