Drawbacks?

Before you get aftermarket bell housing measure EVERYTHING. I did what you are saying. Switching starter from pass to driver. Originally my car had a 904 with custom headers made. I wanted to keep that orientation. ( I have QuickTime bell) offers a gen3 hemi to tkx ( I know you have t56) check your input shaft length! The gen 3 specific driver side starter quicktime bell is too tall.. I had to make a custom input shaft bearing. Reference the driver and passenger side starter bell depths cross reference it with input shaft length. Whatever option you do measure everything.
My opinion cast bell housings are.. eh. I know they work and have better tolerances than older ones but I like my feet..