64, 65, & 66 speaker panel trade

I think the panel you are referring to (on the pass side rear) is where the factory mounted the optional rear window defroster. I don't think the factory ever put a speaker there. I had one there and on the other side when I was in high school. So many people put them in the trunk divider panel, but that made it non-functional to me.