Not sure whats wrong

I have a 1986 300E the color is that dark kind of charcoal color. What happens is every time I wash the car I dry it with terry towels, this is were the problem starts, if I dry it agressivly in other words to a completely dry surface I get this cloudy smearing that once it's there it is almost impossable to remove unless I wet the area and start over. now I can use my liquid Silicon wax to remove the smear and the the surface is smooth to the touch. If I don't dry the car I get ugly water spots all over it, which on a dark car really looks bad.
So what's the matter? To much wax build up, wrong kind of car wash detergent, (Pep Boys special), wax is Liquid Polymer Wax. I use baby diapers and terry towels. It's really a piss off, every time I want a decent looking job I have to completely wax the car! Great work out but...

Thanks,
Chris Campbell
Orange County Calif.
86' 300E