One more thing i noticed is that your symbol rate was just 2.5 Gbps while the bit rate was 10 Gbps.
You must not increase bit rate beyond symbol rate. I suggest you to keep symbol rate higher than bit rate instead. try to manipulate and improvise the parameters accordingly in your design and i hope you will get better results.
Regards