You have so much corrosion around there, the proposition of starting to remove more serious items may result in corroded bolts snapping off or finding other items that are working now but on their last legs and this may lead to giving you all sorts of headaches.

As Gilly has mentioned the pump needs to come off to replace that small hose between engine block and pump. Even though it looks crappy I would potentially leave it there and just get some new bands to tighten it up. You already need a new fan clutch and thermostat gasket.

IMHO you have so much corrosion there it maybe better to let sleeping dogs lie. Already you have removed some items and found they need to be replaced - all in the cause of fixing a small water leak.