Removing dowel steering shaft

You need a roll pin punch to remove it. A roll pin punch has a convex hump in the middle that seats in the pin.
Different than a flat punch. It shouldn't be that hard to remove with proper punch.
But the pin is a steel pin and not a roll pin.