Ball joint removal

So Everything is installed and torqued down to the correct specs. But I noticed that with the castle nut on the upper ball joint came with a thick washer. If I used the washer I couldn't get the cotter pin through the hole because the washer pushed the nut out too far. I took the washer out and it fit. There wasn't a washer there originally when I took them off.

Is it okay to not have the washers? or are they important?

And the proper torque will pull everything into place I assume?