Beanpole,
I've been on a few jobs where the asbestos was being abated. The areas where they were removing it were completely sealed off and no unauthorized people were allowed anywhere near the asbestos itself.
From what I understand of asbestos, the effects are cumulative just like with lead exposure. It's additive throughout your lifetime.
That being the case, I would say any exposure is too much.
If there truly is asbestos in the building, and it's being disturbed, the contractor should be made aware of that fact. If he already knows and has done nothing to ensure the safety of the men working there, I know of several lawyers that would love to meet him in front of a judge.
