318 La with 360 heads

one only has to line up the three factory smallblocks, to witness what is spoken of.
Put the same middle of the road top end and cam on/in each,equalizing the cylinder pressure, and see what happens.
The horsepower follows the torque until she runs outta air. It cannot be any other way; they are inseparable due to the equation that forever links them. You can't have more of one, at a particular rpm, without also having more of the other.

But one thing is true and proven, more displacement .........by stroke moves the torque curve down, and perhaps that is what is meant when using the phrase " displacement builds torque". It's not very meaningful or accurate to say that unless you also have a context, like more displacement at the same bore. Taking that 4.04ish bore from 340 to 367 to 379 to 408/416, you can clearly see the progression of low-rpm torque production.
If you have a tow truck you wouldn't consider a 340. By the same thinking, does an early-A need a 408? What's the right size for a 3400pound car?
A lot of it will depend on the combo and the application.
At 5000 ft, you will need the bigger displacement, ie longer stroke, to be able to process enough oxygen. At 500 ft, the 340 ,ie 3.315 stroke might be adequate. It's always about getting the air(oxygen) in at a suitable rpm, for the application.
I know you gearheads and builders know this; this post is for the OP.

in a nominal 4.04" bore, a 4" stroke is over 20% more displacement than a 3.31 stroker. At 4 inhales per revolution,and at low-rpm, that 416incher is theoretically able to inhale nearly 1.8 more, 340-sized cylinders worth of air on every ONE revolution compared to a 340. That's a really, really........... really, big deal,If you need it.
To process the same amount of air, the smaller stroked engines will have to rev higher; that's just the way it is.
And that leads to hi-stalls and race gearing. Or supercharging.
And that's AJs 2cents