I think the best way to go is to think about who you're talking to first. If it's an american, just say driver or passenger. If they're british or whatever, say left or right. If they're a Star Trek fan or something, or have spent most of their lives at sea as in navy service, use port and starboard.
Or you could just play it safe and be more elaborate. For example, just tell everyone "left looking from back" or "the side closer to the wall in Daytona" (this would be the right side). You get the idea.
Of course, you could always do what I do and keep a micromachine in your pocket so you can SHOW them what side you're talking about
.