Too big of cam?

The cam is just fine, your poor slant just can't breathe and you probably have 2 something rear gears. You need to open up the exhaust, (headers or split manifolds with 2 1/4" pipes and duals) give it some lungs, (4 barrel or twin 2 barrel intake) and add a set of rear gears 3.55 or higher numerically. I go Mustang hunting in my slant powered '66 Valiant with Comp Cam's version of your camshaft with duals, an Offy twin carb intake with Holley/Weber progressive 2 barrels, and 3.55 gears. Just bear in mind that it will cost more to hop up the slant than it will to swap in a V8. It really rubs their nose in it when you spank them with a 6 though.