Regardless of the type of filler primer used I always used 180 to board sand. The faster you take a product off like filler or even primer the more straight the area will become. If you use too fine a grit all the board or block does is ride the humps. You need something that cuts. Then prime with a good 2k primer like PPG NCP primer to fill the sand scratch. I find that people tend to finish sand with too fine of a grit. 600 wouldn't be bad but some people suggest 800-1000 which is not a good thing. You are using mechanical adhesion from this point on and I usually finish with 400. You can actually get away with 320 orbital (not the best way to finish primer) or 400 straight. As long as you are using a good sealer like epoxy.