Ok, I finally got it off. For future reference for anyone else in this situation the lock should be turned all the way to the right. I used a screwdriver to do since the reason I'm changing it is the lock is worn and did not want the tumbler to get stuck. By doing this the Contact Switch plunger is in the up position and I was then able to force it off the rails that hold it. It took some force but I got it.