You keep insisting that you have a Ufer ground, this is better than a ground rod but still not good enough to trip a breaker, the grounded(neutral) conductor is used for fault tripping.
If there is no parallel ground paths common to both buildings, then you should be able to run a 3 conductor out there; however if the AHJ wants a 4-condutor, then he will contiinue to gig you until you convince him otherwise.
I personally find it prudent to say ok and do it the way he wants afterall he is the Authority Having Jurisdiction.
-Mark