They move as is not back and forth towards the push rods making it go and and down but side to side on the shaft the lifters are on and bolted onto the head
You should add spacers so the rockers stay directly over the valve stem. Normally they'll move side to side, but not enough to lose their centering. Solid lifters make noise period. Remember to set the lash as recommended.