On a race car, sure. On a street car I'd rather be able to stop before I hit something, and the larger disks and drums absolutely stop better than the smaller stuff.
I'm sure my current 11.7" rear disks are lighter than the 11x2.5" drum brake set up they replaced, just like the 11x2.5" drums were no doubt heavier than the 10x2.5's, but honestly it's not like you can tell the difference at the seat of your pants. I've been through 3 different rear brake set ups and 4 different front brake set ups on my Duster and all I ever noticed was the difference in stopping power. Maybe you'd notice the weight difference if you started timing things, but if all you ever do is go fast for a 1/4 mile and then stop you don't have to worry about anything jumping out in front of you.