Well, what if a dead short should happen on the far end of the switch leg, or at the light fixture itself? You will need wire heavy enough that won't get fried by the fault current passed by the 20 amp breaker.
Another example of this is the outlet over the sink (near the ceiling) in the kitchen for the electric clock. Such clocks draw only a few watts, and thus around 30ma. Does that mean that you can run 24 gauge wire to that outlet? No, because if a short circuit ever happened at the clock, that thin wire would burn up and the breaker would think it's only someone making toast. The house might be toast....
