I need to remove the cover of the driver's seat backrest to repair a wornout bolster panel.

Is it possible to do this without removing the whole seat out of the car? with the seat installed in the car, I was thinking of removing the arm rest, then removing the backrest cover. Is this possible? Am I missing something? thx