You do bring up a good point: what is it that makes attic air so annoying?
I doubt that you, or anyone else, has ever inhaled vermiculite. The pieces are typically little cubes about the size of a kernal of corn.
If you ever are in a newly insulated attic, you'll be amazed at how little stuff gets in the air, no matter what particular insulation is used. I don't think we breath any insulation at all; I think what really annoys us is simple, ordinary dust that we stir up. That's why old attics are so uncomfortable.