Hello Kamaran,
I suggest to start reading an up-to-date invited paper in coherent PON, first. this will give you a good vision on what is going on in this field. many points you can figure out, where things are. Such as, ultra dense WDM, PON and nonlinear effect on the system performance. Or another point, the single and dual polarization PON.
In OptiSystem, there some examples that you can start with to guide you step by step.
Hope this helps
Mohamed Abouseif