I would have a problem if. I was watching a grand daughter and she needed to use the little girls room. Then! I just saw a man walk in after her in a dress.
What about the man in a suit who follows your grandson into the men's room? Or more likely, the coach who is in the locker room with him every day? Same concerns there, but you don't worry about it because it looks normal.
Maybe the least bad option is to have more "family restrooms" in public places. People who are supervising a child of the opposite sex already use those, and people of non-standard sex could use them without bothering anyone.