Hard part is explaining how the voltage is present on hot and reutrn side of the cicuit with the switch open.
You are measuring the line voltage between two legs minus the voltage drop from the load that the push button controls, likly a coil.
Look at it as a series circuit. The open contact and presume (never assume) for this dissussion is a very resistive load. It will be in series with the coil. Although the coil resistance is high, it will be much lower then the open contact. In a series circuit, the voltage of the circuit will be the sum of voltage of each load within the circuit. In addition the voltage at each load will be inversly propotional to the resistance of each load. The higher the resistance is of a load, the lower the voltage for the load will be.
If we assign a resistance value to each load, R1 being 10 Gig-ohms (open contact) and R2 was 50,000 ohms (coil), the resistance of R2 makes up only .0000005% of the total resistance of the circuit so the voltage will be 99.9999995% of the total voltage. Being that the line voltage is so high the voltage drop is less across R2. Although it appears that the voltage is the same, when the contact is open, you are actually measuring is the line voltage minus the voltage drop.
A digital voltmeter will likely pick up the voltage but not a analog meter due to its high resistance.