AFB Tuning

I started working with Carter 4bbls in the 1970s. AFB, AVS & TQ. Also Edel later.

I have never used one or tuned one that needed more pump shot than using the outside hole on the pump arm, the least amount of pump shot. Later AFB & AVS,TQ & Edel used a 13/16" plunger. These are 4cc. Std Holley is 3cc, big pump is 5cc. So Carter pumps are quite big.
I say it may be a pump issue. The AFB is an older carb and maybe the plunger is old or hard and you are not getting the full 4cc. I need to change my plunger every couple years when I notice a difficult start. It is a easy try, without removing the carb and doing other adjustments. I would like to note, being your car runs fine, except for that miner glitch, right down all your starting points before any adjustments are made.