Yeah, the design of the cam follower on the rocker will allow pretty smooth wear on the cam lobe, making it sometimes difficult to diagnose as a worn lobe. It's been my experience that when the hydraulic actuators begin to go bad, they generally peck loud at startup, then smooth out and quieten down.
Check the rocker pivot/shaft also; any loosness there will really tap.

As stated, try to find a later 103 cam and rockers; but do not mix older rockers/cam with new ones. The newer ones are surface hardened.

DG