Should I be worried?

I would get a magnetic oil drain plug.
If you really want to know what in that oil, save a sample and get a free oil sample kit from blackstone labs and send a sample to them and they will tell you exactly what is in it.
http://www.blackstone-labs.com/