Last year, I put new tires on the front of my car and it pulled to the right so badly that I literally had to "hold" the steering to the left as if making a left turn. Swapping the front tires helped, but the problem was still there. (I understand that this is called "radial pull".)
The supplier replaced the tires for me and all is okay again. You may have to either replace the tires or use them exclusively on the back.
Further, consider the amount of drainage "crown" in the road on which you are test driving. Some are worse than others and can cause the car to drift toward the edge.
Doubt that you damaged your frame. They are tough.