Note I said "at the circuit entry into the sign". I agree with you, either move the disconnect to the conduit entry point or change where the conduit enters to make it next to the switch, preferably in a separate enclosure within the sign.
This could be as simple as a 4x4 deep ring that covered the switch terminals and conduit entry with a blank cover on the other side.
I believe the purpose of the disconnect is to make the interior of the equipment safe. I am not making a proposal but I do think a label over the disconnect enclosure saying something like "line voltage present with disconnect off" is a good idea.