Hi Maha,
Have you studied the OptiSystem tutorials regarding how to cosimulate OptiSystem with Matlab? If you learn about the format of the OptiSystem signals, then it will be quite easy to implement the filter itself and not the subtractor. For the main program, you just need to use two “if” commands in Matlab.
Regards