Yes, you can just remove the air-pump belt, but the EGR will still operate if you don't disable it. To do so, you can either plug a vacuum line to it, or disconnect it. I agree with your idea of removing it all and cleaning up the bay. You can even get a Euro front exhaust manifold to avoid covering the hole after removing the EGR.

You can remove everything in the diagram with the exception of #9 and #39. When you're finished, #9 should have one vac line direct to the intake manifold and one vac line direct to the retard side of the distributor. The advance side of the distributor should also have now a direct line to the intake manifold as well.

__________________

Don't Chrome them; polish them