Major cam problems!!!!

For those of you that don't know the whole story.
I have a rebuilt 360 with magnum heads. This engine only saw dyno time before I bought it. It sat for 2 years before I installed it. Upon my initial startup everything sounded good, after I installed my exhaust I could hear very loud rocker noise so I investigated and found one lobe had damage and others looked like they were on there way out. All other indications showed no other damage to the engine.
I installed a new cam and lifters. When I broke it in I started to notice some valve train noise from the same side as the first time. Today I took the top of the engine off just to check and make sure things were O.K.
Bad news. I now have 2 useless bump sticks. Although the cam itself isn't damaged bad it is damaged and a few of the lifters are showing wear, enough that they are junk.
When I installed the new cam I used liberal amounts of cam lube and after I had installed the rockers I even coated the lobe's again. I primed the oiling system and was getting oil to all rockers. When I took the valve covers off today there was oil trapped on all rockers so I don't think it is an oiling problem. I used Rotella-T 15-45 and GM EOS as my break in oil.
The engine took about 15 seconds to really catch good on the startup and then I ran it for 13 minutes at 2300 rpm before it started to get to hot. 220 to 230 degrees and I shut it down. The next night I ran it for another 10 minutes at 2500 rpm. During both of these runs I would take the engine up to 4000 rpm for a few seconds and then let it go back down to 2300 to 2500 rpm.
I believe I did everything right. The valve stems look good and all valve heights check out good. The valve springs are rated to .525 lift. The first cam was a Comp Cams XE268H and the second was a MP 284/484 purple shaft. With the 1.6 rockers the first cam had a total lift of .509 and the second cam had a total lift of .516. I checked for coil bind as best I could and did not find any.

Has anybody got any idea's? Could it be a geometry problem? Would going to a roller cam solve this type of problem?

I don't know much about cam theory and any help would be good.

Jack

360 +.040
magnum heads 1.6 rockers
10.5 : 1 comp
M1 intake
no cam