The basic time zone division is based on 360 degrees / 24 zones giving 15 degrees of longitude between time zones.
I assume that where the natural boundaries fell near to state lines that somebody decided it made sense to make the boundary detour a little. If the line cut through the middle part of a state, then in many cases it follows county lines. The boundary between central and mountain zones in Nebraska does that, except in Cherry Co. in the north where it just divides the county in two (very big county!).
I think Sven's suggestion about minor detours to account for population centers makes sense. The boundary you mentioned in Indiana follows county lines and puts the northwestern corner into CST. I imagine somebody thought it made sense for Hammond, Gary and the other urban stretch of Indiana along Lake Michigan to be in the same time zone as Chicago. By the way, doesn't Indiana also have the peculiarity that those counties in CST use daylight savings in summer but the the rest of the state doesn't?
Looking at the boundaries on a map, it does seem as though other features were sometimes followed. Look at the Florida panhandle and the boundary follows the Apalachicola River, (flowing south from the Chattahoochee and Lake Seminole at the GA/AL border, itself the EST/CST boundary).
The far western tip of Texas is an interesting peculiarity. At some point in history did somebody there decide that El Paso was far enough west from the rest of Texas that it should go into the mountain zone?
The situation is much simpler in the U.K. We're so small that the whole country is in one time zone!
