I would double check the stab ratings on the main panel that is feeding the sub. Some panels limit the maximum current ratings of loads connected to a single bus stab.
Also the use of #4s falls right onto an ugly bit of NEC 310.15(B)(6). #4Cu is permitted to be protected with a 100A OCPD when it is used to supply the _main_ power feed of a single residence. #4Cu does not have sufficient ampacity to be protected at 100A _in general_, but for the specific load described in 310.15(B)(6) it is permitted to do so. This means that #4Cu is _not_ sufficient for the feed to a 100A subpanel.
But there is yet another exception, which is that the feed to any subpanel in a residence is not required to be larger than the main feed. So I think that using the #4Cu as you describe will be okay until such time as the main service is upgraded, at which point you will need to protect the subpanel at less than 100A.