On a Dutch forum a member who used to do the maintenance for a fleet of W123 taxis back in the days, said that he replaced the bearings of the vacuum pump every 200,000 km as a prevention. The bearings don't cost much and fatal engine damage could be avoided. If you only replace the bearings, and not the entire rocker arm (which is expensive), be sure not to deform the "fork" when you press the bearing in. And press on the outer ring of the bearing, so as not to put force on the bearing balls.

Alternative is a new rocker arm or a new vacuum pump.

It is difficult to say whether your vacuum pump will fail, there is some evidence that too much play on the intermediate shaft will cause vacuum pump failure, see this thread:
http://www.peachparts.com/shopforum/diesel-discussion/204187-what-particular-causes-vacuum-pump-failure.html