I think you'll be fine with the torque of the 408 with that cam, But, the 2 things i'd change are the gears/converter, I'd go with at least a 28-3000 stall & 3.91s & run a 28" tire. Even 4.10s would be nice for a street bruiser. I ran 4.56s in a 370ci smb in a full bodied 73 Swinger with a 2800 PTC 11" & it was a monster, the cam was the XE284H, it had stock J heads with 10.3 compression, Just for kicks i installed some 3.55s & it killed it literally.
Bottom line IMO with that cam, the gears/converter will be your weak area.