Ductwork is sized based on the capacity of the ac/furnace unit, not the efficiency. You need to move a certain amount of air to deliver 4 tons of cooling, for example, less for 3 tons, more for 5 tons. The amount of air you can move is based on the number and size of ducts.

Most homes have lousy ductwork - undersized, poorly designed, poorly installed. In a properly designed system the air should flow silently. If you have rumbling sounds, rattling grates, whoosing noises - then the ductwork is likely undersized. That's the most common problem, as HVAC contractors cheap out on every nickel they can save.

You want the ac to run continuously to keep the house comfortable on the hottest day. It is more efficient - and more comfortable - to have a smaller ac running more hours/day than a larger one which cycles on/off to maintain the temperature of the house. So if the 4 ton unit isn't working very hard this summer, then downsizing to a 3.5 ton might be the right thing to do. The longer run times of the smaller unit will do more to dehunidify the house, so it will be more comfortable at the same temperature as with the larger unit.