Yep, I would.
If you have loose chips in that housing you really need to get 'em out. Chips may be imbedded in matting splines and could possibly damage your new axle. Worse yet, if a large enough remnant gets pulled into the gearing and locks it up at speed, it could put you in a bad way. Not saying that will happen, but you don't know if you don't check.
