I do not believe you stated that the exterior panel was locked or capable of being locked but this may have prompted the inspectors decision
230.92 Locked Service Overcurrent Devices.
Where the service overcurrent devices are locked or sealed or are not readily accessible to the occupant, branch-circuit overcurrent devices shall be installed on the load side, shall be mounted in a readily accessible location, and shall be of lower ampere rating than the service overcurrent device.
The main lug panel might be a good choice.
Charlie
