Hi Umar,
Since I don’t have access to your simulation file, I can only make some general suggestions that I have formerly made in other topics. You can use DCF to reduce the dispersion effect. As for the launched power, you can increase it to the point where the nonlinearities kick in. You can use optical amplifier after the transmission line, but not more than the fiber attenuation.
Some advanced modulation formats such as OFDM have shown better performance especially against dispersion and can be considered to be used in PONs (especially in downstream, with upstream being OOK for example and using remodulation techniques), but since “cost” is an important factor and should be taken into consideration, it is not scientific to say which modulation format is “better”. Otherwise, all the researchers would have focused on one or two modulation formats.
Regards