Dear Jonathan,
The theory of the MUX/DEMUX filters can be found in the datasheet of the component. You may access it from the pdf file (OptiSystem_Component_Library.pdf) in the documentation at the following location or from the Help button of the component properties popup window.
There are different filters in OptiSystem ranges from AWG, thin filter, FBGs, etc. that can be used for MUX/DEMUX implementation.
C:\Program Files\Optiwave Software\OptiSystem 17\documentation
Regards,
Ahmad