I don't understand the rational that argues:

One wire size if I put the outbuilding service at one place and a different wire size if the service is elsewhere.

Certainly the diversity of load at the outbuilding is identical. And the current demands on the wire are identical.