I would suggest you, start with the two tutorial volume pdf documentation available after installation. the go through the samples available of your interest then you can better understand the working o the systems. further study the journal papers that has been implemented using OptiSystem to have a better idea.
Regards,
Dhiman