I believe sections 300.3 (B) for all the conductors in a circuit to be in the same raceway, auxillery gutter , etc. would apply.
Also section 250.4 (A)(5) for a low impedence path could be a problem with the hot coming from one panel and the grounded in another.
I don't believe it is worth the possiblity of having problems. Just extend the groundeds to the sub panel and be done with it.
