I have a similar thing at my house, several locations that use the audio from the entertainment center in the living room. I distributed line level signals and have separate amps at the user locations. Then they have independent control. Your immediate problem is the combined speaker impedance. If you merely switch them in and out of a common drive, the volume will change as will the load on the amp. I am not sure how you will do this without a pretty special amp.
