Hello,
Is it possible that you send me your project layout and the Matlab code to check it out? Please email them as a zip file to ahmad.atieh@optiwave.com.
The only thing that I see in the code that you are missing to define the type of signal in the Matlab code (optical, electrical, binary or Mary). You need to match that with the signal type at the input and output ports of the Matlab component in OptiSystem project layout.
Regards,
Ahmad