Good point, I might give the rubbing/tapping methods a try.
The edges are a but rough because my bushing driver always shifts a little bit when I drive a bushing and makes a little ridge on one side. Once its driven in I go back and "shave" off any ridges with a razor. I realize that that is not optimal and probably contributed to my tight bushing. It is what it is.