Those are not swirls that is wax residue. Use a quick detailer spray to remove it. You are using too much wax. What happens is, especially on oxidized areas, the paint is grabby so when you attempt to remove the wax it doesn't quite come off and is difficult to remove and leaves a residue. On smooth well prepped paint Gold Class is easy on easy off, you need to get your paint to that point first.

If you are doing this by hand then try a wax with a cleaner. If you like Meguiars then try Meguiars #6 or Meguiars Cleaner wax or Meguiars Color X wax. These will help eliminate the oxidation and the grabby feel of the paint. Also try a clay bar before this step as suggested above, you need to get that paint smooth!

Then throw Gold Class on at the end.

Good luck.