You are an enabler! You are allowing this man to take advantage of you. He is not doing it to you; you are allowing it to be done to you. No one can do anything to you that you don't condone. If you don't like the situation fix it or stop whining about it. You know what is wrong and you know how to stop it.

As Niki ads put it, Just do it.