I think you should do a load calculation as a first step to see if you actually need 96 kva. (240V x 400A)
The only practical way to minimize voltage drop is to run HV cable up to the house as you suggested, however it might be just as cost-effective to pay the PoCo to bring it to the house, if they will.
