Hello Gurinder, The system you have attache d is almost fine,But at the constellation diagram of first channel (As you are using WDM system) is more noisy than at channel 2(you considered only two demux).This is due the reason of the different delay time of two receivers.Also you have used direct SMF of 100KM without any dispersion compensation.You have added a EDFA after 100KM fiber to mitigate the effect of attenuation.I have made several changes in your system
1)Use dispersion compensation
2)Power of transmitters 0dBm and Local oscillator with phase adjustment,always remind as you increase power non-linear effects also increase.
3)Symmetric compensation with DCF
I have used 2 loops each of 50Km SMF and 10KM DCF followed by EDFA’s.You can increase this distance further by increasing power and using non linear compensation like optical phase conjugation.Tell me if you require any assistance regarding some good research papers on this topic.Hope i have made my point clear.Good luck for research
I am attaching a file by doing these modifications