24 AWG -- Certainly adequate for the sort of relay coil currents you'd need in this application (i.e. < 250mA). It will have to be low-voltage switching, as CAT5 isn't rated for 240V.

I have to ask: Why CAT5 for a proposed dimmer system? Did you just happen to have some spare?