That sounds like a bad exhaust mani gasket. When the engine warms up, materials expand.
When it's cold, pop the hood and listen -- have someone rev it a bit. You should be able to isolate it.
Do you hear a ticking as if it's valve-train racket? Sometimes, the car can make a chugging noise when at cold at idle. These are my experiences...