you can start with “Matlab & Optical System Cosimulation” using Matlab Call OptiSystem.
It can be downloaded from:
Home » Resources » Downloads » OptiSystem – Hot Topics
This example demonstrates the use of Matlab® software to call OptiSystem and calculate the bit error rate after the OptiSystem simulation.