It's not the best way to do it, but there's no reason why you can't. 110 hardware is rated for CAT5 performance at a minimum and at all terminals.
Personally, I'd terminate all cables on separate four-pair clips and just manage my cross-connecting accordingly. That way you aren't committing to an "all or nothing" wiring infrastructure. Flexibility is the key to doing it right.