When to worry about engine noises?

Are you using a pressure reading (manual) or electric sending unit on the oil pump gauge?
If there is a concern try a different gauge, preferably one that reads the pressure directly from the port with no fittings besides the one necessary to attach it to the engine.