How much bearing damage is too much

I would change it it looks almost galled with that material on it.