First of all..it's your butt. No one else, in the end, is responsible for you. Heck, you'll be lucky if they send cards and flowers!
So, if you are uneasy, it's time to re-consider. Yes, you may suffer- be ridiculed, get fired, whatever- but it's time to be a man and make your own call.
One advantage of being in a union (no- I'm not getting political- stay with me here!) is that you have someone, in the person of the "shop steward" who can address your concerns without your painting a "bulls-eye" on yourself, and who might not even be a parrot for the foreman.
General comments aside....from your description, this sounds like a situation where fall restraint equipment is required. I am certain that a visit from OSHA will see a change in practices at that jobsite.
