Hust for ha'-ha's, what is the voltage line- line when this happens? if its steady at 240 then its almost guaranteed its a bad neutral.
My former employer had a problem identical to this. the problem was found to be a bad connection between right at the transformer bushing... One thing that tipped it off was we did an amprobe check at the main ground where it connects to the water line, zero amps was flowing.. If it were to have been a break elsewhere in the neutral you might see current flowing in the ground, taking a path through the water piping in the street then back through another ground in another building or through some other path back to the center-tap bushing of the transformer...

A.D