You have to be careful not to raise a burr when you do that so you don't score the crank...
A burr in the front side will score the crank, a burr on the back side will make the bearing tight and cause more resistance for the crank to turn or even lock the crank up...
It doesn't take much for a bearing to get tight, I've seen a small speck of dust or string from a cotton glove lock up a crank due to the interference it causes with the bearing on the front or back...