Radiator/trans cooler is a remote possibility, but it would be my second thought.
Your observation of your vac line showing sponginess/swelling, along with hard shifting and rough running engine [possible oil soaked plugs causing misfires], along with vac leak , would indicate a bad modulator diagnosis...but that is just my first thought..the proof is when you take the line off and see..pretty simple to check.
With your info, Looks to me most probable cause would be modulator diaphragm is split..very common.
If they are not too bad, there would not be enough smoke to brings one attention to it, but it would be something to check for..

__________________
A Dalton