Dear Saiful,
Please check the setting of the output port in the Matlab component in terms of electrical, optical, binary or M-ary. Also, not that it has to match the definition of the port in the Matlab code.
Please refer to the Example Library located at the following link for more information on how to setup the ports in the component and in the Matlab code.
C:\Users\USER NAME\Documents\OptiSystem 17.0 Samples\Software interworking\MATLAB co-simulation
Regards,
Ahmad