Get some stones and sanding rolls. Burrs are great for removing big chunks, but lack finesse until you get good with them. Stones and paper are easier to control and also blend better because of the built in flexibility. Stones typically won't load up with steel like they do with AL. You're going to want them anyway to finish the ports, might as well get some on hand and see how they compare. You might wind up saving some time or frustration.
Long tapered paper rolls are available and can work great around the guide bosses.