Can't figure out an electrical draw?

OK let's get this straight.

You have eliminated your draw down to ONLY the yellow wire on the stereo, right?

The next step is to determine if it is enough draw to really affect the battery

So you need to get/ borrow a meter and hook it up in place of your test lamp and actually measure the draw. Depending on your meter, it may display in decimals of an amp (ampere) or milliamps (thousandths of amps), I.E. 1000 ma is same as 1 amp.

DO NOT DISCOUNT the fact that there actually could be some sort of electrical failure in your stereo that is causing a more than normal draw.

Is your battery going down? How long does it take to do so?

How GOOD (old) is your battery? Have you had it "load tested?"

How is the charging system, is it keeping the battery "up?"