FYI, the largest naturally occurring vein of asbestos resides in Fairfax County, VA (where I live) and no ones moved the homeowners out yet.
So long asbestos is left alone and not interferred with, it is a pretty inert compound. The problem is when people start digging into it or disturbing it in some other way.
