Did this a couple of weeks ago.
Your Idler Arm assembly is exactly the same as mine.
There is a dust cover on the top (big washer with drip-edge facing down), but it is not attached to the welded tube in any way. The bottom just has a washer under the nut.
You may be looking at the actual bushings, which are metal on the top.
I found the best way to remove them was to use a punch or chisel that was small enough to go through the hole in the upper bushing, and drive out the lower bushing from behind. Next, I drove out the upper bushing from behind. Prying would probably work, but might not be as easy.
I removed the rear exhaust manifold, though I have heard you can unbolt the engine mounts (and exhaust mount at transmission) and lift the engine enough to get the needed clearance instead. Your model may not have the pre-cats that really get in the way, though. Without the pre-cats, I bet the bolt would come right out!
Some folks have suggested cutting the bolt and installing the new one upside-down. I decided I like the extra insurance of having gravity on my side if the nut were to come off for any reason, however.
Note that the "spacer washer" on the top goes between the arm and the dust cover, which was certainly not my first guess upon reassembly.
1986 300E 5-Speed 240k mi.