Tired seals inside the transmission are the most likely cause of this problem. I had my 380's transmission rebuilt at 232,000, primarily because of harsh (manual) 4-3 downshifts. It helped, though I must say the problem is not entirely solved.

Another area to look at is the integrity of your driveshaft's flexible couplings and the center bearing support. Replacing those items on my SEC improved the 4-3 downshift dramatically - no more harsh clunking. I've almost finished the same job on the 380 and it'll be interesting to see if that helps.