Hello sanjeev kumar
As of now i am not an expertise in coding but regarding the simulation of matlab code in optisystem you just need to keep in mind the signal types and sampled lengths. Else the matlab code follows the similar logic as in the matlab software itself. The major difference in the code for optisystem is we define ports with the same name as in optisystem and that is e.g InputPort keeping in mind tha caps as well. Rest for coding a demodulator i am sorry I don’t have much knowledge of coding. But since u have coded modulator it should be easy for you to design demodulator. Best of luck!
Regards
Burhan