The placement of the Matlab component depends on the functionality that you desire. So, if you’d like to calculate BER then you place it after the detector section. Again depending on the code you wrote if you include filtering or other processing then the placement may vary.
You need to make sure that the input and output ports of the Matlab component are properly set in the component properties window. Also, you need to handle the signals properly in the Matlab code to allow correct transferring of data between OptiSystem components and the Matlab component.
I recommend viewing the examples in OptiSystem Example Library located at the following location to get better understanding.
Please note that you can’t post OptiSystem project on the forum. You may email me the file as Zip file to firstname.lastname@example.org.
C:\Users\USER NAME\Documents\OptiSystem 19.0 Samples\Software interworking\MATLAB co-simulation