Hi Sasha,
The problem in your 4-QAM system is the fact that you have not set the M-ary threshold detectors parameters according toa 4-QAM system. I believe you have copied these components from an implementation of 64-QAM. Please take a look at OptiSystem coherent 4-QAM sample to see they should be set for your system.
Regards