mufflers or no mufflers??

Im no expert mechanic by far, but I do know that your current exhaust system sounds pretty good flo wise. 2 1/2" pipe would be ample for a mild 360 I would think, & 40 series mufflers flo descent as well. Not the best flowing muffler in the world, but they are the only thing I will ever run, good flow or not.
Could be your cam. What cam are you running? Some cams dont work well with restrictive manafolds.

The only thing that I can think of that would be bad about running straight pipe is
1: the noise & you already know that.
2: old timers will tell you that running straight pipes will burn up valves over time.

Other guys that know alot more than me will chime in, so be patient.

Welcome to the site!! You will like it here.