Now that you have Matlab program for the DSP component and you want to co-simulate Matlab with OptiSystem, all you need to do is make adjustments to the format of your Matlab codes and use Matlab component to do the co-simulation. For this, I suggest you go through tutorials and files available in OptiSystem samples regarding Matlab co-simulation to learn how to make the necessary changes.
Regards