Hi Ajay,
Couple quick questions: was the constellation diagram that you attached measured at the end of optical link or at the end of radio link?
What type of radio channel did you use?
If I’m not mistaken, the coherent detection is used for QAM-64. In this case, my main guess to explain such amount of errors is big amount of phase noise, especially in large dynamic range. To fix this problem phase noise cancellation (PNC) circuits are used (you should be able to find papers about it).