Generally speaking, exhaust noise is exhaust noise, period. If anything, a leaky exhaust manifold and compressor should call themselves to attention even more so at higher speeds.
I'm more inclined to believe the problem lies past the manifold towards the muffler or connections within. Performance mufflers are "tuned", for a rumble at idle, but quiet down at low speeds. There may be a leak somewhere...
2009 ML350 (84K) - Family vehicle
2001 CLK430 Cabriolet (71K) - Wife's car
2005 BMW 645CI (124K) - My daily driver
2012 Mustang V6 (60K) - Daughter's car