Depending on the build, that stem release button may be located on the right side of the switch rather than the bottom. In either case, If you'll remove the 1 screw that holds the fuse box in its clip, The fuse box will move a little. Thus you'll have more space for hand/button pushing.
And by the way, no need to push the button when installing the stem. Its tapered point guides it in.