Hello Subhash,
Could email me your project as a zip file to ahmad.atieh@optiwave.com to debug it? It is difficult to debug the issue.
From the constellation graph, the modulation is not 16QAM. I see only 8-spots, which may indicate 8QAM placed circular.
Regards,
Ahmad