We're in a society that seems to be increasingly polarized and (aside from your example, but reaching to the subject you chose) that's a label that carries so much negativism it's often used as to hamstring the speaker.
Perfect example: criticize Obama and you'll often hear the racist label attach...and good luck climbing out of the hole.
As to the invite...well, it takes all kinds. Maybe his narrow-mindedness (and the reference to a dish to share) means "black" attendees would bring "black" foods and he doesn't care for chitlins, watermelon and whatever else the stereotype suggests?

While his attitude sickens me (and you wouldn't find me in attendance at the function as a result), I'd hate to live in a society where people are told who they must invite into their homes.
The perfect irony would be a white classmate whose spouse was black. Of course the classmate would likely be "uninvited" anyway.