Sounds like a bump steer issue. Check it by working the suspension through it's travel and measuring the ends of a couple of sticks hung horizontally on the outside of the tires. Easy enough to do on a Mopar by backing off the torsion bars and having the car jacked up. The fix is shimming the steering box bolts and slotting the idler arm bolt mounting holes to move the relay rod (GM term, sorry) up or down until you've minimized it. Don't forget to at least tack the idler arm bolt washers to the bracket to hold the setting after you've found it.
Lack of caster can contribute to wander. A bodies with stock upper arms and bushings are noted for a lack of caster, could be a problem with your B body too. Moog sells offset upper bushings that will help you get a little more set in. Not unusual for the caster not to be the same side to side either and they can help that too. Last ones I bought cost me about double compared to the regular ones. Not as affective as tubular uppers (most tubular arms have about 4-5 degrees built in) perhaps, but a lot cheaper to try.
