I think you absolutely did the right thing. I'm a non-union guy and I would have done exactly what you did. Like some of the guys have said, there are some bad contractors out there that could careless about your well-being. But on the other hand, there are some good ones who do know how to run a business and do care about their employees and take good care of them. The catch with working for someone good is that you will be expected to work your butt off and not make any mistakes.

Good luck finding a new contractor to work for. There's plenty of work out there and I hope your search for work will be short. Good luck.