I think the only thing holding it then is the bearing, unless you forgot a screw or something. I would just keep working at it. The rubber cover goes all the way around the bearing so if you try to pull it out it's gonna tear it up. Maybe try some pb blaster or something, maybe the bearing is grabbing the shaft.