I didn't want to hit it too much with emery cloth for fear of making it too much smaller and having the bearing spin. I ordered the Timken bearings and seals and FelPro gaskets.
It'll be fine as long as you just concentrate the polishing in the area of the imperfection. Just don't polish where the bearing and lock ring press on and you'll be fine.