Vibration at 75mph. Bad hub?

The inside of the passenger side hub did have some roughness to it like something had been rubbing at some point. I took some 320 grit and smoothed it out a little bit but not enough to where the race wouldn't fit in there snug. The original races were a pain to get out, I don't think they were loose in there. The passenger side one could have spun at some point but it was tough to get out of there.

I think what I'm going to do is pull the drums off my girlfriends 67 Valiant and throw them on my car and see if the noise goes away. I was confused when I took the first drum off and saw that the race was seated a little ways above the lip inside the hub. Then I pulled the other side off and it was the exact same way. When I pressed the races back in, they seemed to want to stop in the exact same spot, which makes me believe thats just where they're supposed to sit.