Hi Ina,
I have went carefully through the paper you have attached.
first of all, I do believe OptiSystem can model the proposed system. All the required components are there in OptiSystem.
just make sure to mimic the all components correctly in the paper to get similar behaviour of the output curves. I see it is written the three hybrid modulation techniques is reported in [5, 6] and then the author call the output signal orthogonal modulated signal. So the three data; Data1, Data2, and Data3 could be extracted in the output port.
you can try and also send us your design to try to check it with you.
Hope that helps,
good luck
Mohamed