I've seen more of the old rock wool blown-in insulation than vermiculite. Although, when I do see this stuff in the attic, I know I’m going to have to build dams to keep the whole joist bay from pouring down all over me when I cut in a ceiling box.
I think the fine dust that you can see swirling around with your flashlight in a dark attic when you disturb this stuff is what they are concerned about. I usually wear a mask, but I’m not sure it’s the proper NIOSH filter for this product. When I was younger and didn’t know any better, I probably inhaled my own body weight of this junk, since we didn’t know enough to take any extra precautions back then.

They used vermiculite in gas log fireplaces too. The new non-asbestos replacement product looks just like the old stuff to me, so I can’t really tell them apart unless I see it in the package. They recommend sweeping as opposed to vacuuming as the preferred method of cleanup as an added precaution.