Dear San,
the BER will be minimum and Q-factor will increase only when your system design will be efficient one i.e. you have to adjust the parameters of the components, use some advanced modulation scheme such as DQPSK or QAM at the transmitter side and also go for OFDM which is found to be more immune to error. Regarding the fiber channel you can use optical amplifier such as EDFA to compensate for signal attenuation, then you can go for dispersion compensation i.e. you can use use dispersion compensation fiber for reducing the dispersion that will help you to increase the Bot rate of the system. At the receiver side DSP block you can use which is consist of many filters or algorithm such as DC blocking, Normalization, Low Pass Filter, Resampling, QI Compensation, Dispersion Compensation, Nonlinear Compensation, Timing Recovery, Adaptive Equalizer, Frequency Offset Estimation etc. that will help you to get a better result in terms of BER and Q-factor. Moreover it will be design specific such as which detection scheme you are using (homodyne or heterodyne).
Regards