What you are missing is an intermediate shaft
This is driven by the camshaft, and the upper part of it engages the distributor, while the lower part engages the oil pump
(The flat spot you see on the bottom of the distributor goes into this shaft)
So when you swap distributors, the intermediate shaft stays in the engine
If you ever change over to a high pressure oil pump you may want to replace that shaft, but at this point I wouldn't worry about it