I was thinking somehting like that might be a way to do it.
Do you tap all around to drive the new one in, or put something across so th drive from the center?
If the ground off bearing is exposed i use a hammer and go to town. One hit then across with another. Back and forth and criss-cross. Similar to driving them out. Then i clean the bore again.