The shop I was at use to do all our quick jobs with 320 on the D/A then paint. Never had problems, Not that I would do a car I cared about that way.
Yup you are right. I actually always used 320 orbital on collision work and it works just fine, but for restoration work I would prefer to block sand.