Could possibly be a wiped cam lobe as well. Personally I would pull the valve covers before you drive it further and have a look. Excessive valve train play on one or more will warrant deeper investigation. If it has wiped a lobe running and driving it more will spread metal through the engine and cause more damage. What type of cam / lifters are in it, are you using proper oil and additives?
When my Mopar 509 can wiped a lobe it was similar to what you describe.