Thanks Damian Marek for replying
The total band i am working on is 10 GHz as the sampling frequency is 20 GSample/sec. So but the bandwidth of each user is 2.5 GHz which can be anywhere in the GHz band. So if the sub-carrier frequency is around 1 GHz there is no dispersion but if the sub-carrier frequency is around 8 GHz there is large value of dispersion.
I did not a LPF to the PD but there is a LPF after electrical down-convertion (demodulation) to the baseband and it is set correctly because if i turn off the fiber dispersion in the optisystem the BER is very small
So the problem seems to be in the dispersion.
attached are the system setup and the optical spectrum when the electrical subcarrier frequency is 8.75 GHz
