Head rests

Is there a trick to removing the headrests from a bench set or am I just being too careful with it. I can pull them up and they click into a higher position but they wont come out. Do I just need to pull harder or is there something holding them in that I could potntially break?