I have had good luck with clamping the piece to a sheet of plywood or plank, then clamp the whole thing to your drill press table, and use a holesaw with some cutting oil, and take your time. Run the saw at a slow speed and use just enough pressure to get it to cut, use oil or even water to keep the heat down.
