Niko,
I believe that you are correct. If one does a load calculation for a typical residential service, even for a large house, the maximum 120-volt loads are very likely well below 100 amps. Then if you assume that all those loads are on one phase, which would never happen, a #4 copper would certainly be plenty large.
However, when one tries to do something that is out of the ordinary or unusual the typical reaction from other tradesmen or inspectors is at least surprise or disbelief. I do agree with you that the neutral can be sized to the maximum unbalanced load if it meets the other requirement for minmum size of ground.
Be prepared to show your service calculation if you want to win that arguement with an inspector. A more common size for a neutral in a residence is two sizes smaller than the phase conductors. This provides plenty of capacity for future circuits and avoids wasting your time arguing. I suppose the bottom line is the value of your time to do the load calculation and discuss this with an inspector versus the amount you save in wire costs. If the cable length is significant, then it might be worth your while to have the discussion in advance with the inspector. Good luck.
