That doesn't sound right. I'm no carb expert, and never had a double pumper either, but you're basically idling on the secondary circuits, which sounds like you have an internal leak or something out of whack. Was it rebuilt, or is it new, or...? You should be able to get them somewhat balanced.
Check this:
"If your car or truck has a camshaft with considerable duration (and as a result, a very lumpy idle), adjusting the idle mixture screws may not help very much. There’s an easy fix: adjust the secondary throttle stop screw. Holley four barrels like this 750 cfm 0-3310C have a small set screw located beside the passenger side throttle bore shaft bore (on the base plate of the carburetor, so you’ll have to flip it over). By turning the screw clockwise approximately 1-1/2 turns, the throttle blade on the secondary will open slightly. This allows a rather large amount of airflow through the carburetor without increasing the “window” seen by the idle transfer slot. At this time, the primary throttle blades can be closed, which tends to enhance the sensitivity of the idle mixture control."
from
Quick Tech: How to Properly Set up the Idle on Holley Carburetors - OnAllCylinders