Please find the attached papers. One is for QPSK modulator, the other is for coherent detection. The principles are explained very well in these two papers.
By the way, the sample you attached is in old version. If you upgrade it to the latest version OptiSystem 12.2.1, you will find a better sample with DSP in it.