These 12V lights are typically 50W. If you have only 3 lights you have 150W. On the primary side at 120V, the dimmer has to deal with just over 1A (including the transformer). On the secondary side at 12V, I would guess there's going to be more current than the dimmer can handle.
Andy