Hello Aseel,
the Q-factor could be improved when you receive a signal with higher power and less noise. The noise could be due to different types including ISI, interference, amplifier noise, beat noise etc… The noise could be either at the transmitter, receiver or in the channel. It could be a result of dispersion or nonlinearity effects or both in the fiber. So, in summary you need to minimize the noise and improve your signal to noise ratio. You may use better detectors or coherent detection to improve the SNR.
Regards,
Ahmad