Would be nice to know what car.

Diesels with pans usually have acoustic damping material on the inside surface of the pans.

I've been driving my 87 300SDL without the pans for about 3000 miles and it's noisier with them off. I don't think I'm doing any harm by leaving them off until I get around to patching the tears.

Sixto
91 300SE
87 300SDL
83 300SD