Cams good, I checked it. I also saw some black marks by where the intake runner meets the head. I sprayed some carb cleaner on there and the idle went up. I think I have a bad intake gasket. Better than a bad cam I guess.
Always better to have a bad intake gasket...unless you were looking for a reason to change the cam out ;)
DarTT may be correct though and I can't recall if you're using the same intake you were before.
Stupid question: All the bolts are torqued down to spec right? I know SB Fords have issues with the intake gaskets settling and the bolts effectively loosen up...causes issues very similar to yours.