Sorry, I glossed over the fact that you have rigid conduit from one panel to the next. That is your fourth wire so you want the neutral and ground separate at the second panel.