Need Help To Remove Inner Bearing

That's the bearing race and needs to be replaced at the same time as the bearing itself, it's what the bearing rollers run against. Yes it will be tight as it is an interference fit so it doesn't move around. When using a punch to tap it out move the punch to a different spot 180 deg from the last hit otherwise you'll jam it and have a hell of a time. Installation of the new race should be done with a hydraulic press not hammer and punch.

Terry