Dear,
You can download many coherent systems from optiwave website for example 100Gbps PM-QPSK
I’ve seen the design you attached,I didn’t inspect it but from the constellation image,i think that there is phase error between the Tx and The Rx.
I used power splitter after the CW laser and used the same laser for both the Tx and Rx but i found only two constellation points but received correctly but without Group velocity dispersion.Also the RF spectrum analyzer results seems not good.
I suggest that you follow the Optiwave examples attached in the software and start with it,You can find the Optisystem xx samples with the optisystem run in the program files.
I hope Damian can help you more.
Best Regards
Abdallah