shooting primer

Silicone is a bad contaminant as well as any wax etc. When I worked in bodyshops,wiping it down came first. I always thought of it this way.

If there is any crap on the car and you dont remove it first,it gets on your grinding wheels dics sandpaper tools etc etc.

Maybe you just need a faster solvent? Or more flash time between coats.