You do not need to take off the switch. Rather, you have to remove the large screw that tightens the door handle. That is on the other side of the switch.