bigger rear drums

i have seen alot of threads on here for disc brake conversions. i have seen little talk of the back brakes on here. i figure to change over to the big bolt pattern in the fall. in doing so i figured i would upgrade my back brakes while i was at it. i have the factory drums on my 72 now 10 by 2 1/2 inch. can the 11 by 2 1/2 inch drum setup fit with the 14 inch rallyes? i figured i look for a 70 thru 74 imperial , newyorker or fullsize wagon, and grab the whole back brake setup. i will get new axles from mosler. will this work? are there any drawbacks with this setup?