I agree with Scott, the instability of the time base circuitry, is the most likely cause. This can vary by quality and tolerances of components. Line garbage can be a factor. Many are quite accurate.
This reminds me of one of my "Sea Stories". On the nuclear test site, in the Pacific, we used WW radio time standard signal for synchronizing of time, by injecting this signal into our master timer. We were encountering an interference due to receiving two signals at different time intervals. This was traced to the fact we were receiving them from each direction around the world, and they were of different distances, and a resultant time. We designed a circuit to exclude one signal, and only amplify the other. Talk about accuracy, I prefer my Mickey Mouse Watch.
I understand the atomic decay, time base clock is the most accurate at this time
