Replacing column shift lever

I've driven them out. It shouldn't take much effort with the correct size punch. Supporting it is a good idea. I'd more concerned if you were pounding on it lengthwise - which could damage it.