Forged steel 340 crank

Wednesday I "intercepted" my crank from the machine shop. It's been degreased and ground .010 under and is awaiting balancing and chamfering of the oil holes, but here's the thing; This crank, while being forged, and thus highly desirable, is really sloppy. There is forging flash all over the thing, sharp edges, and very rough counterweights (on the sides). My purpose in "intercepting" it is to clean all this up with a carbide bit, files, Roloc discs and whatever else will work. Note this will be done only on the rough forged areas; I'm leaving the machines surfaces and journals for the pros. My question to any racers or pro builders out there is; how carried away should I get? I realize the smoother the crank, the less oil drag will be generated, but am I making this a bigger deal than it really is?
The cast crank I took out, while metallurgically inferior, is a much "cleaner" piece........

I've attached a pic of the crank before it went in for grinding.........

crank 340.jpg