Should I be worried?

There's almost always something left somewhere in the engine during assembly. The silver as everyone says - additives, cam lube, bolt lube, nothing to worry about. the gunk - if everything's fine don't sweat it. Seen much dirtier engines that went on to be fine. That's one of the reasons you drain the break in oil quickly...