I replaced the ball joints. They were rusty, but didn't seem too bad... Anyway, the problem is still there. It is a bit less severe though, so the joints must have been somewhat tight. There isn't anything else in the front end except the idler arm, so I guess that's next.
I will update when that is done...
Thanks again for all the help