Ok....so next time it does it, rig up a jumper wire from the terminal that the yellow wire normally plugs onto, to a ground. I usually squeeze a ring connector on one end, and take a screw holding the starter rely off, clean the bottom of the head of the screw, and then put the ring connector on the screw and tighten it back down. Put a push on female spade connector on the other end, and plug it onto the lug you unplugged the yellow wire from. Then see if it will start with the key. That will tell you whether it's the starter relay or the neutral safety switch.