Of course, there's probably just a teensy, weensy bit of air leakage out of the building through the fan [Linked Image]*, so insulating around it probably isn't all that important.

(* Well, OK, if the fan works properly, there's a huge amount of leakage... [Linked Image] )