Don't remove the speaker. I don't know if it would help anyway, as you'd only be able to push on the left side through the hole.

Buy or make some hooks. I made a pair. You need less than 1/4" of a hook to grab the slots. Or make the hooks bigger and longer, and get the panel from the rear.

Put the steering wheel all the way down and back before you start, and remove the key. Pull only on one side at a time to release the clips. It will move (with steady but quite firm pressure) about 1/4" or so when it unclips. Once both sides are released, the panel comes out easily.