Well, to improve spectral efficiency, we have to either opt advanced modulation schemes (higher ary QPSK, QAM etc, MSK), WDM/DWDM or reduce the spectral width of optical source (laser to reduce dispersion) or sometimes use of coherent detectors also help to improve the spectral efficiency. Here, I think Dhiman wants to use DSP to cancel noise, improve receiver sensitivity so that overall data rate can be increased.