Another brick wall trying to remove steering wheel collars

Please bear with me...I am somewhat confused.

You have the steering shaft, then a sleeve, then this disk which sides envelope the second sleeve.

I need to pry the second one up to find the pin..would that not be the same as pushing the outer one down ?

I can see where the outer one is important.

Is the outer sleeve attached to the inner one in any way other than friction fit ?

I fear I am damaging the outer sleeve and not moving the middle sleeve...

Ian.