The first questions are "is this fed off of a sub panel? Is the neutral bonded there?"
You might be seeing the voltage drop in a 3 wire feeder.
If so that is where I would start.
You should know "ground" is not always "zero volts". That is why we have article 682.
