Removing Tappets

Well crap. I have the cam out and my initial idea was correct they free float in the block then. I guess a dowel and a hammer will fix my problem.