I've never replaced the idler arm bolt on a W124 300D so I don't know if the exhaust has to be dropped. You can probably figure it out for yourself. Look under the car and see if there is room to lift the idler arm bolt its entire length straight up from where it is. It's got to be 15-18cm long. You might be able to wiggle it out if the bushings are worn, but you won't wiggle it in with the new bushings.
86 300SDL (due soon)
81 300SD (for sale cheap)