Actually, what you mention is a sore point with many neighbors.
In Reno, even an author, writing books at home, and communicating through the internet...with NO visitors, NO shipments, etc... would be in violation of the ordinance.
It sounds as if you are in compliance with your town's ordinances, so... let the neighbor fume! Look at the bright side...some day, he'll want some work done- and you now have a ready excuse (I cant't, lest some fool turn me in for doing business out of my home!).
Such folks are miserable fools, who delight in spreading their unhappiness around. "No Este Basterdum Carburundum"