Should I be worried?

Depends on how much, I had a little when I rebuilt mine, after an oil change it went away.