Should I be worried?

Was the oil pan perfectly clean to start with?

I'd run the old oil thru a coffee filter and see what you get