When I got the car I did all the normal fluid and filter changes (oil, ATF, fuel filters, coolant, power steering, and so on).

I consider the differential fluid as being a needed fluid change when buying one of these cars, but I could not get the fill plug loose. Because there were so many other problems I needed to deal with, I put off this task.

Is it harmful to drive the car in this condition, assuming the diff fluid needs replaced, and is causing the noise? Or is it more of those "its ok to drive but sounds annoying" situations?