I agree with the strut rod theory. Someone outside the car should be able to see the tire jerk toward the rear at the moment you hit the brake pedal.
The following links will give your more info:
http://www.peachparts.com/shopforum/showthread.php?threadid=8458&highlight=Caster
http://www.peachparts.com/shopforum/showthread.php?threadid=12712&highlight=Caster
David