Without even looking too closely at your numbers I can tell you that a rotary phase converter, unless it is a true motor/generator (a single phase motor coupled to a 3 phase generator) will not give you true 3 phase power. Any of the ones I have seen use the two existing "phases" as A & B which are at 180 deg and create a third phase. How do you get 120-120-120 deg out of that?
I think your buck/boost transformer is seeing this phase descrepancy, which may change under load, and not provide the proper correction.
