Stupid question

Have you taken the mirror down and looked underneath? Some have a plate that simply holds pressure against the ball. If this is the case with yours you can remove the plate and flatten it some or add a small shim to create a little more resistance.