How do I remove this piece?

the bearing has a rubber on so it wants to hold on but my bearing did not break nor did the rubber tear. i think i actually may have laid it on a bench and used a rubber mallet.