As you noticed, we don’t have a multimode demultiplexer in Optisystem, we will investigate setting up this type of capability but we have not determined the target release.
Are you looking for an idealized model to separate out the modes or a physically based MM Demux?
For the idealized model, one possibility would be to perform some Matlab Cosimulation:
We have a component called “Mode Selector” in our “Receivers Library/Multimode” directory from which you could extract out each transverse mode. Knowing each mode profile, you could then process this with Matlab to perhaps convert this to a non-multimode optical signal and then you could use our standard demuxes.
If you are looking for a physically based model, our FDTD product manager mentioned that there are some designs that could be simulated using OptiFTDDTD. For example: http://www.opticsinfobase.org/oe/abstract.cfm?uri=oe-21-15-17904