Hi Adil
Actually, it is not fully clear to me your question and your system. However, you can build any subsystem using matlab/C++ -based component and include it into your OptiSystem project. there are some examples on how to make this integration.
if you can clarify your question more we can discuss and try to get a roadmap for your project.
thanks
Mohamed