How do I remove this piece?

The snap ring will generate a burr at the ring groove over time. Pull the shaft up a tad. Use a small file and emery cloth to clean up. Housing with bearing should slide up and off. Never was a press fit there.