Just a wild guess ... but think for a moment ... how does a dimmer work? Usually by letting only part of the sine wave through. Where does the rest go? Back into the grid.
Would the unused part of the wave cause additional heating, or other problems, inside the transformer?
IMO, dimmers and transformers are a problem looking for a place to happen. If you're controlling a resistance type load - incandescent lights and heaters - then perhaps an autotransformer is what you really want. If you're controlling a motor, then a VFD is called for.
If it's a DC load, then control it AFTER the DC is created. Just my thoughts.
Actually a dimmer simply chops the sine wave at a certain point. The unused part of the wave is simply unused. It doesn't go anywhere any more than the power hitting an open switch goes anywhere, Chopping up DC is harder than AC and you certainly don't want resistive control if wasting energy is a concern to you.
I haven't seen any issues with having a dimmer in front of a Malibu transformer
Why? A transformer will simply smooth out the spikes and give you an average voltage out. There is nothing uglier than the output of the chopper in a switching power supply and they feed that into a transformer. I guess I could take a scope to the one I am using and see what the waves really look like in and out. You might get some strange results going into an unregulated supply feeding a radio or audio amp but if this is just going to feed LV incandescent lights I don't see the issue.
Here's my guess: A dimmer is an electronic device that has a voltage drop of maybe 2 volts. On a 12 volt circuit, the max voltage left for the bulbs is 10 volts, or about 90 percent. The same 2 volt drop on a 120 volt circuit leaves 118 volts for the bulbs, or about 98 percent. On a 12 volt circuit the lights would be dimmer.
These 12V lights are typically 50W. If you have only 3 lights you have 150W. On the primary side at 120V, the dimmer has to deal with just over 1A (including the transformer). On the secondary side at 12V, I would guess there's going to be more current than the dimmer can handle.
It's always easier to get forgiveness than it is to get permission.
Andy and TW may be on to something. Although the current itself might not be a problem (a 50a device is not that more expensive than a 5a one) the power dissipation will be. The output triac will drop ~.7v (silicon device). Since power is amps times volts and the .7 volt dropped in the triac is fixed, doing this on the 12v side will mean the device has to shed 10 times what it would on the 120v side for the same light output. I am still curious about what these wave forms look like. When the mob of kids and grandkids go home I will take a look.