Dear Dhiman,
If you try using DSP and improve your signal reception, that will not help improve the system spectral efficiency. The flaws with the optical sysem will remain as it is. I suggest instead of using DSP you try to improve the optical system such that the overall system spectral efficiency can be raised