74 Swinger brake CLUNK!

The wheel spins freely. If the car is parked and I start it, but go right to neutral, it will roll if on an incline. When I shift it into gear, as soon as I start to go forward, you get the “clunk” and it is almost as if I am pressing the brake pedal.