Passenger-side mirrors are only useless if you are like most of the population and point it straight back. You don't need to see the door handles (they're still there, trust me) and you can already see straight back with the inside mirror. Point the mirror further out so you can see the next lane.

I use the passenger side mirror and would never consider removing it.