Has he outright refused to pay, is it a disagreement over cost or quality, or is he claiming to be a little short right now? I have often wondered the same thing. I believe that the materials used are still yours until you are paid. If you have access, start to remove them (unless he has a gun). Did you have a signed agreement. I try to get one on anything over around $500.00.
