Hi. How do I remove and replace the seat switch to a 300SD?

The back-forward direction is not working and I've got a replacement switch. I've removed the two mounting screws and the upper two screws to the door handle. I expected the switch to pull out with a little bit of pressure but it's not coming and I hate to pull (can't!) harder. Does the door panel need to come off?

Thanks!

Randy