Put a voltmeter on the coil of the starter (btw, if it doesn't have an overload attached, it's just a contactor) and verify that the voltage to the coil is removed as soon as you press the stop button. If that checks okay (my guess is it probably will) then the contactor is faulty. The motor is going to be connected to the Normally Open contacts and what's happening, I think, is that the contactor isn't immediately returning to its "normal" state when the coil is de-energized.
I never tried to open one up to service it, in my line of work components like that are always replace if defective. Since you have nothing to lose, try taking it apart to inspect the internals.