What you suggest certainly seems reasonable.
Theories are fine, what you need now is data. Start working your way upstream with your meter. Check the feed to each panel in turn. Eventually you'll get to the buildings' main disconnect.
Ultimately, you'll kill power at that main disconnect, to check your incoming feed with no load from the building. If your voltages are suddenly OK, isolate that transformer and check it with a megger. A short within that transformer could be overloading the PoCo feed, and bringing the voltage down.
