Massive caster variance Side to side

At rest, what effect does the brake have to do with anything?
The wheels do turn somewhat depending on the caster. Locking the brakes stops this.

I’m not how much of a difference it makes.