Probably just stuck if you've got the allen out.. give it a pry or a hit with a dead-blow hammer.

If the hub is rusty they can be tight.

M