1967 Dart 273 2bbl odd fuel delivery issue at idle

Has anyone observed the following behavior?

I have a clear fuel filter ( ya I know I know)
When the engine is at greater than 1000 to 1500 RPM the fuel filter is completely full
When the engine is idling the filter will slowly drain till it is completely empty.
The engine rpm will reduce until a squirt of fuel comes into the filter (but only enough to wet the bottom of the filter)
The RPM will increase slightly
then fall off again
Then another squirt
The RPM will increase slightly
then fall off again
and continues that way till the RPM is raised to 1000 or greater

I suspect it is the fuel pump, (it was replaced in 2011 but sat for 4 years without running) There is no issue at speed or higher RPM which seems counter intuitive for a bad fuel pump

Carb was rebuilt 1 year ago, engine runs a bit rich at idle.
Engine was stock rebuilt in the 80s

Thoughts?