Here's what I've done many times - buy those glass/tile bits...but in 1/8" - 1/4" diameter...they cut Real fast...drill a number of holes in a circular pattern for the larger diameter you need (I've done this for 4" diameter)....once you have your numerous small holes cut you can easily knock out the center section with a hammer, chisel...yes you have a ragged hole...that's what caulk, etc. is for...to keep the conduit off the ragged edges and patch up the hole nice and clean.
