Check and adjust your valve clearances if they have not been done. What may be transpiring here is as the valves warm they elongate faster than anything else. So you have perhaps a partially open valve after warm up.

This same valve when cold and not expanded probably seals well enough to let the engine idle fine. Well worth checking if the idle is solid when the engine is cold.