As long as the valve lash is somewhere near correct I'd drive it given your seemingly gentle use.

I'd think your noise may be more camshaft related than bushing wear. Mechanical cams have a gradual initial ramp that takes up lash before the major lift occurs. As this ramp wears out the lash is taken up more rapidly and can lead to noise.

You might have noise from another area, have someone very familiar with engines to have a listen.

For the record I have not worked on your engine type but, have built countless engines over the years on a professional and personal level. My shop has been closed for some years but I still have my own projects going along.