I sure wouldn't want to use on a day that you are not busy!
In order to be consistant and fair with each customer, you should charge a fee to everyone, everytime. You wouldn't want two neighbors to discuss why one of them was charged a minimum fee and the other wasn't. One of those persons is going to be very mad.
I suggest charging a fee to everyone, but do not attach a time limit to it. Call it a travel fee, show up fee, service fee or anything else, but not a "1 hr minumum charge". When you charge everyone the same fee, it will make you more consistant (and thus easier to prepare invoices), and make you more money. An extral $100 a day in "travel charges" really makes a difference in your bottom line.