318 Rocker Clatter
I have had a bit of grief with Comp lifters myself recently (on a Buick 350). I am living with it until I get real bored and pull the intake. Basically 2 of my lifters bleed off at real low RPM. I am guessing you may be suffering the same fate. But here are some other things to check if you want to make a semi-scientific diagnosis.
Bent pushrods: It happens, and if your 318 ever suffered a cam gear failure, bent pushrods are almost guaranteed.
Worn rockers/shaft: If the shaft had grooves worn into it and you took it off the engine, forget ever making it totally quiet again, even if you get all the rockers in their original location. I wouldnt think wear on an 80,000 mile engine wouldn't be too severe as long as the oil was changed somewhat regularly. If you put your finger on the rockers at idle and push sideways a bit does the noise stop? You can carefully make a rocker a little "wider", but dont get carried away. If only two have more wear or play than the rest, I would start asking myself why.
Collapsed lifter: As I said above, I have had problems with Comp lifters right out of the box. Try to isolate which lifters are ticking, and have a hepler crank the engine with the valve cover off. Stop the noisy valve at full lift and watch. Does the lifter bleed down in less than 30 seconds? If so, your lifter is junk.
Wiped cam lobe: Yeah, this is the ugly one nobody wants to think about, but it happens. Pull the valve covers and make sure you are getting the same lift on all valves (intakes may be differnt from exhaust, but all 8 of each should be the same). You will find the above mentioned collapsed lifter real quick doing this. No need to bust out a dial indicator, a steel ruler with a couple sharpie marks will give you a basic "big picture" type diagnosis.
Exhaust leak: As you said, it sounds like an exhaust leak. Are you absolutely sure it isnt? If pulling spark plug wires makes it change notably, suspect exhaust.