It's not so much the speed that may be the problem.
Picture yourself driving from the passenger side when you want to pull out to the left to pass a big rig on a two-lane road.
It's no longer an issue of pull out just a bit to see if traffic's coming. You'll have to have over half of your right-hand-drive car over in the opposing traffic lane before you'll be able to see what's about to run head-on into you.
I've got a Mopar buddy that lives in Oz and he's told me that this is a BIG problem when driving our left-hand-drive classic Mopars down under.