removing steering column

I need to remove the steering column soon. I was looking at it yesterday and noticed that where the pin is located the hole is larger on one side than the other. Which direction does the pin come out. I did briefly tap it with a hammer from both sides and it didn't seem to move