If you're mounting the switch more than 18" above the ground, you should be okay at that distance. Don't forget to use IMC or GRC and the appropriate sealing fittings for the conduit run.
And I would suggest you run it past your local AHJ just to avoid having to change it after you're all finished with the installation.