How can it multiplex four controllers onto two controller ports? Would it be possible to extend it further with three devices?
EveryNES and NESdev Wiki may help you sort out some of the details. The two joypads connect to the NES through a form of serial communication. The controller sends a synchronized 8-Bits across the #7 pin connector.
The value of the button that is pressed is encoded within 8 bits that are communicated and synchronized with a clock signal sent across the #2 pin.
When the NES Four Score or NES Satellite adapter is connected, the adapter sends a 24-bit stream across both controllers' #7 pin channels. The first bytes on each line correspond with controllers 1 & 2's buttons. The second byte across both lines corresponds to controllers 3 & 4 and the last byte is an "ID" byte used to detect the adapter's presence and verify that none of the preceding bits were deleted by a known glitch that occurs during sample playback.