Just remembered::n my '81 Ram 360 I swapped out the original thermoquad and 4bbl intake for a stock 318 intake and 2bbl carb just to get better fuel mileage. Works great:::been on there for 17 years and guess what? "severe port mis-match" again.
But the port mismatch is your favor because the intakes ports are smaller than the heads ports. If it were reversed, then I would say theres a problem. The air and fuel crash into the head and then try and go into the port.
Your move is a smart one. The reverse would not be.