my heart dropped to my knees.....

They make a tool for removing the races, but they can get pricey. I've always used a screwdriver on the backside and a hammer. The spindle nut you're talking about is called a castle or crown nut.