Update:

Thanks for the opinions. Unfortunately, I have not found the source of the noise.

dpetryk: My car doesn't have hydraulic pump...
Arthur: The damper is in good condition. No vibration.
mpolli: The noise starts after the first 30 minutes... And the sound is not as smooth as the air pump makes.

Now I can dulicate the noise with 30% success rate. These are how I can duplicate the noise.

- The engine must be on or above the working temperature. (>80C)
- The noise comes only when the engine idles, the transmission engaged(D/R/3/2/1), and the brake applied.
- Push the power window button (UP) when the windows are closed.

I letted my wife push the window button to find out the noise source from outside... failed. I can hear the noise outside, but it is VERY subtle.

Since the noise is far more noticeable in the cabin and it is from VERY driver side(almost the driver side door), I think the noise doesn't come from the engine itself or something on the engine. It's from behind the left side of the instrument cluster or under.

One more thing. When the noise is noisier than usual, I can feel vibration, too. The vibration comes through the steering wheel and driver side floor.

Any opinions?