I am a student in mechanical engineering working on a personal project for which I need to use a 3-way switch triggered by a photosensor (one sensor for the on-position, and a different one for the off). Could someone point me to some resources that would help me to understand how this is done and how the circuitry is designed?

You will not be using a regular off the shelf IR detector unless you add some extra logic.
If you used the detector to pick a relay and then used a "D" flip flop it would work. You could also do it all with relays if you want to go "old school".

I'm sorry, but could you clarify that some? My knowledge base is in mechanical systems - I could design this to be entirely mechanical but then I run into an issue when I am machining the parts for the thing. They would end up being overly complex for the system I am designing. Electric control seems a more streamlined and efficient method. But, again, I know next to nothing about the subject, so I need it expressed to me in layman's terms. Or if you could point me to where there are some resources for me to study, I would appreciate that as well.

OK do you want one detector to always be for "off" and the other "on" or do you want it to toggle whenever either is triggered?
Considering your situation, a relay solution may be the easiest.
This is really beyond the scope of this BB but I can send you a private message.

http://www.mpja.com/Opto-Interrupter/productinfo/18028%20OP
Are you going to use opto-interrupters like these? You could use the outputs to steer the input of a schmitt trigger circuit, biased half way between the on and off thresholds. Same thing with a comparator with hysteresis. Two analog options. You can drive an R-S flip flop or D or J-K flip flop with preset and clear pins as digital options. Then there are the relay options. So many simple ways to do it, depending on the voltages that you wish to work with and your actual application.
This would be better posted down in the technical area.
Joe

Wow, that is some real deep stuff there Joe!

You lost me at the Schmitt trigger......

I'm with Greg; relays.

A Schmitt trigger is just a switching device that responds to a slowly changing input with a clean turn on or off at a certain point in the input slope.

With photo detectors, an op amp is a pretty good way to accomplish this, using a feedback resistor to square the output up.

I would modify Greg's description to mention 2 different points on the input slope. For instance, with a 10V supply, it might take >=6V to turn the gate on, but <=4V to turn it off. 2, 10K resistors could be placed in series across the supply to place 5V on the input. Momentary switches across those resistors, (probably with their own series resistors to prevent shorting out the PS), would be able to switch the output back and forth.
I would also agree with you both on relays if our OP is using line voltages and large gaps. I picture those photocell circuits that ring the bell when you walk into stores. 2 of those and 2 relays or 1 latching relay, would be all that's needed for what I think he's trying to do. But for small gaps and logic level supplies, I picture a flip flop with preset and clear driving an opto, SSR, or relay via a driver ckt.
Joe

In the thermostat I use for my spa, using an LM324 op amp for the trigger, my feed back resistor is pretty big (1m pot) so the on and off points are pretty close together. +/- < 1 degree f on a thermistor sensor (automotive water temp sensor)in a Whetstone bridge with two 10ks on the far side.

We really need to know more about what kind of photo sensor he is using and what the outputs are. I agree a latching relay is probably the lowest part count solution if he can drive it from the sensors and he wants one to always be "turn on" and the other one always "turn off". You can get latching relays that operate around 1 ms so if this is not really something where things are screaming by, it should work.

I've used off the shelf photosensors on the TEST mode for things like gate buzzers and driveway alarms. They shut off almost immediately when no motion is sensed.
If you install 2 of them, hook both the switch legs up to the primary of a 'Touchplate' transformer, and the secondary to a Touchplate (latching) relay to control the load it should take care of the situation.

...and it would use all unmodified UL listed parts

The former is how I have it designed. What I am designing has a reciprocal motion - I will need each sensor to do only one thing. Sensor one will turn on an outside circuit when activated, and sensor 2 will turn off that outside circuit. They will never operate the other way around.

The latching relay may be the simplest solution. It has 2 coils. One picks the relay, the other drops the relay. Getting from the photo detector to having the current to operate the relay coils will depend on which one you chose.
I am sure you can find something that would operate them directly.

Look into P&B KUL series latching relays for what you need. Potter & Brumfield was gobbled up by TE Connectivity, which used to be Tyco, so don't let the different mfg. names throw you off. I don't remember you mentioning your desired operating voltage yet and that really does impact the best fit of many possible solutions.
Joe