exhaust pipe diameter for a mildly modified 340

Long day....just got in and see lots of chatter. Great stuff guys!
OK... the mufflers are the restriction and back pressure has no effect!
I have watched most of the Engine Masters videos but they are usually for hi hp builds through open headers for the quarter mile.t
I do like a quiet exhaust, Rumble. Ultradyne tech spec'd the cam and said headers would produce no real gain in my build. Engine was built to give good street performance and will never see high rpm.
So...how much gain or loss could you notice driving on the street from 800 to 5000 rpm. 15 hp...20 hp?