Replacing Ignition Switch in 73 Duster

It does look like there is a kind of sleeve on there, but it definitely does not just lift off or slide off easily. I tried with a vice grip a bit but it bends some and I didnt want to damage it or anything. Here's what I'm trying to get off: