Welcome to one of the issues with leaf spring rear ends. Any distortion or damage to the spring mounting along with normal factory tolerances (or lack thereof) can create this issue.
The only way to prevent it or keep everything centered is with a panhard bar or watts link. But before...