The clunk when you downshift without vac is a good sign. It means your tranny has been working properly. When you hook up the plastic vac line to the modulator again it will provide vac to the tranny for downshifts and it should downshift smoothly.

Unbelievable that this is exactly the same design used on my 1981 300SD and it is so good that they were still using it fifteen years later.