Dear Adree,
Did you try to run the examples in OptiSystem Example Library located at the following link?
Which version of OptiSystem are you using?
C:\Users\USER NAME\Documents\OptiSystem 17.1 Samples\Software interworking\MATLAB co-simulation
Regards,
Ahmad