Coulnd't one mitigate any variability in the dyno tests by running multiple dyno runs under each scenario and averaging the results? Just thinking out loud here, since I didn't see any of the tests.
Yes, that is correct. The best way to do dyno testing is A-B-A testing where you run a baseline test, then test the new part, then re-run with the baseline. If the baseline repeats then you have some confidence that the test was valid. But hardly anyone ever does testing like this since it takes a long time and costs a bunch of money. I've run several hundred dyno tests over the past 20 years and I've never met a customer who was willing to pay double to go back and verify the baseline test. Once you install the new trick intake or carb and see the power increase nobody wants to put the stock parts back on to verify that the baseline hasn't moved.
I've done some baseline testing on my own engines and the results are usually mixed. I did a bunch of dyno testing last year with one of my 470 engines where we tested parts all day long and added power with every test. It felt great since everything we did made more power. Then the next morning the power was back to the same place as the previous morning. We never did figure it out for sure but it appears that the correction factors don't properly correct. So as the weather changed during the day the power changed. In our case we thought we were picking up power from the modifications but evidently it was just the correction factor changing. The next morning we were back in the same spot. I've run the same engine with zero modifications at different times during the day and have seen more than 20 hp difference. So that difference is due to the air changing even though the dyno numbers are "corrected". So when someone shows me a dyno test that "proves" something I know better. Proof is fairly elusive on an engine dyno unless you have a super expensive setup. I think some of the NASCAR boys have climate controlled rooms and that is probably what you need to have to see the little stuff. You also need some sort of big electric motor to spin the engine in order to measure friction. Not very many guys have access to that kind of test equipment.
