Hi Gaurav,
as Ravil, I also think that using Matlab is the best choice. I remember an example from a previous OptiSystem release (probably 11) on how to implement the DSP algotithms with Matlab, but I cannot find it anywhere on Optiwave site…maybe you will be more lucky!