Your program should work in Matlab, but you should change its format to make it work in OptiSystem. For example, in your currently written Matlab program, I cannot find where the input is received from OptiSystem (you are going to replace it with OFDM component, so the input should be the output of coder).