I wouldn't expect frost to be a problem if the conduit extends below the frost line.

The temperature change of the portion below ground is narrower.

Frost can heave inches in certain soils. A simple expansion joint won't solve that problem.