If this GC has never had anyone bill for this than he is uninformed and is using uninformed sub's. Or they are just including it upfront. I'll bet they are just taking it out of their profit and rationalizing it with themselves that they figured it. I used to get intimidated by these types, but not anymore.
The only time recently I have eaten this cost is when I forgot to mention it upfront (maybe once in the last several years).
My old boss closed his EC shop and went to work for a big commercial GC running multimillion dollar job and saw how things get done. He then went out on his own and I did a couple of buildouts for him. There was no question at all about who pays these. Even his contract excludes these fees and say owner is resposible. Now this is coming from a climate of strong-arming favors, buying back bids, chipping small percents off expense after the job is sold, strickly to inflate profit by whatever small amount he can. If he'll admit it's a legitimate charge, then it is.
I also had another guy that would sell sunrooms all over this area. He would complain that he was paying $xxxx.xx per year for me to be allowed to work in all these different areas. I just laughed and told him I'd mail him a check for half if I get another job in that village before the year is up. He got the point that it is a job expense to that specific job. I also always gave him the option to get someone local, and that would shut him up too.
The only one I chaulk up as overhead is my local city that holds my license. I need it to keep my license current and I do my highest percentage of work here.
Permit fees are not usually included- if for no other reason than various government agencies keep changing, or adding to, their fees. Such fees are also, when you get down to it, the responsibility of the property owner. I (In a "turn key" operation, this would also be the GC).
So how to resolve a dispute. Simply don't call for an inspection until you've been paid. The permit is in your name- only you can call.
The GC you describe is, at the minimum, unprofessional. You're probably better off working with someone a little more, well, competent.