If it's not an open house situation, as others have noted, it's by appointment only. Perhaps nothing more than a poor choice of phrase by the agent, who might instead have said, "I'm sorry, showings are by appointment only, please contact your agent" or something like that.

I also keep thinking of the "Seinfeld" episode focusing on nuance.

What did the person emphasize?

Did he say, "I'm not comfortable showing you the property?"

Or was it, "I'm not comfortable showing you the property?"

Or perhaps, "I'm not comfortable showing you the property?"

If the second, I can see how one might hear racism. Otherwise, I think it's a fair chance it was just a poorly-chosen phrase under the circumstances.