The starter is bolted to the belhousing through holes in the block flange, so yea, it's gotta come out! It's only 2 bolts and the wiring, you'll probably encounter much worse as you remove and install your motor!
Sorry, no no connection to the block. But you are right about one thing, it's only two bolts. I cannot imagine trying to shoehorn / align an engine back on a bell with the starter in the road.