We might have a few misunderstandings. I said that I agreed with @WAYNE0's comment that he sands with 180 or 220 before the primer on the fiberglass, then I added that I usually take it further to 220 and 320. I have never had a problem with high build adhesion on fiberglass sanded with 220 or 320. I am a bit confused where you say, "I never go finer than 80 prior to paint". Surely you don't mean that you sand primer with 80 grit and then paint over that, or that you block sand primer with 80 to smooth it out and look for flaws. Prior to epoxy over bare metal, I sand the bare metal with 80. Prior to paint, my final sanding of primer is 400 wet for non metallic colors and 600 wet for metallic.