You said that there was tool steel down in the bottom of the hole. How thick/big is it. The reason I ask is that I'm a machinist and have had to deal with these problems before. Tool steel is brittle and it will break if you smack it with a dull punch and hammer. If we ever had a broken drill bit or even a carbide drill bit broken off in a hole we would use a dull punch and a hammer and bust it out of there. But it depends on how thick/big the piece is??