Green Axles.

Okay . In the two pictures I have slid the pin from one side to the other. If this button was in there, that pin shouldn't move, right?

View attachment 1715712165

View attachment 1715712167
.
No button to remove. You can grind the end of your axles to get the little bit of play you need.
In your carrier that centerpin is designed to wiggle back and forth. That's because with tapered axle bearings the axles are shifting a little bit from side-to-side going around corners and such.

But since you have green axle bearin your wiggle is very limited.