>>but technically I do not think that applies.

For a 4 wheeled vehicle, you can implement castor with a vertical swivel axis and/or a tilted swivel axis. As long as the projection of the swivel axis on the road surface is ahead of the centre of the contact patch, there will be self centring action.

For the OP, I would jack the car up, and temporarily disconnect the inner track rod ends. Then, try to swivel the wheels by hand - if, for example, you have a ball joint which is seizing up, this could hinder self centreing action. By disconnecting the inner, you can also assess the friction levels in the outer track rod ends using a similar method.