Uninformed consumers is always a problem for someone trying to do things the right way.
It was always a problem in my Architectural business too. With my real estate rentals I advertise on Craigs list and get a lot of phishing emails but they are pretty readily broomed out. They pretty much never come to look at anything and give the sort of stupid comments like your fellow did.
In thirty years of practicing Architecture I had two contractors who tried to physically intimidate me and one or two who came close to doing so. I was in the situation of having the upper hand since I controlled their money supply so it was pretty easy not to get angry. As far as being influenced by it, I always figured not to give in to wrong requests and let the chips fall where they may. If threatened with a gun or knife I would have given in at the moment but gone to the police immediately later.
If one is a crook these sorts can perhaps be manipulated but I never was inclined to try to work with idiots. Life is too short.
I can think of two clients who I accepted on large projects knowing they probably were going to be a pain and a few who started out fine but turned to the dark side when things progressed to a point where they thought they could finish the project without me. In cases like this I gritted my teeth and finished the job correctly so it would be no excuses right and endured their difficult natures for a couple of years.
I have also had a couple of business partnerships where my partner went over to the dark side. In this case I bought them out if possible, the other again, gritted my teeth and finished the job doing my best to protect myself.
Early in my Architectural career I had a partner who went over to the dark side. I responded with legal action and we wrangled for a year or so until there was no money left in it. I avoided legal action as much as possible after that.
I had one business partner I wanted to assault myself.
Thank goodness I did not.