Watt work, I admire your devotion ... it's pretty late!
The comparison of employees to independent contractors strikes to the heart of the matter. Independent contractors are NOT employees. They carry their own licenses, bond, and insurance. That's why they're called 'independent' contractors.
So, the issue of an employee performing contracting raises all manner of legal issues, all by itself.
If the legal requirements are met, the employee IS absolutely free to do as he wishes when he's 'off the clock.' The employer has absolutely no say in said employees private time. This has been reflected in numerous legal rulings, where employers attempted to exercise control over such time ... even to the extent of directing religious, political, and family activities.
That sort of thing went out with the 13th amendment.
Now ... I suggest you take your "I get to limit the activities of another contractor to your local contractors' board, and see what they think. While you're at it, you might ask your attorney about 'restraint of trade' laws.
The 'understood conditions' you assert border on the illegal; they're certainly immoral.