As you describe this, I fully expect that the problem is the starter solenoid. These are available as a Bosch repair item. I have had mixed results replacing only the solenoid.

If you were working on a 240D I would recommend replacing only the solenoid because you can remove and replace the starter in a 240D in a matter of minutes with no frustration at all. On the five cylinder, however, removing and replacing the starter is a challenging operation. I would replace the entire starter, and I would probably wait until it laid down for good.

If the problem were the battery, the starter would turn slower, not delay in its engagement.

Good luck,