I've always been told not to do it, but I can't see a theoretical reason.

There IS a code reason ... namely, that the single pole breaker is almost certainly not listed for more than 150 v. line to neutral. Indeed, even the two pole breakers in that panel ought to be specifically listed for the use .... though I admit that they usually are not! Most two pole breakers are not listed for use with a 'stinger' either.

Another code reason is found on the panel itself. Reading the fine print on the label, again, the panel is listed only for two pole or three pole circuits with a delta service. What you're supposed to do is use this panel to feed a separate single phase panel.