Call his bluff. Heck, be nice, offer to give him a lift to City Hall, the Contractors' Board, etc.
NEVER give in to terror, bullying, or extortion ... it only begets more of the same. Giving in also lets them think they're right.
And, yes, dump the job. If you feel threatened, by all means make a report with the local police ... nothing will happen now, but it will support your view if things get bad. Ditto on asking for a restraining order. If he wants to meet ... meet in neutral place, with witnesses, etc .... say, a trade association. Or, your lawyer's office.
