I think that a call to a lawyer would be in order. What happened if the house burned down and someone had died and it was related to the work done. The town thinks you did it and you would end up in a corner trying to prove you didn't do the work even though you pulled a permit for it. I say file something against the person for your own pertection down the road
