Hello Nabila,
If I understand you correctly, you wanted to use the PAM receiver of the example (PAM fiber link w Threshold Detector.osd) with QAM system. Please note that the output of the DSP is a normalized output with QAM levels, not voltage as the PAM decision system. you could use the PAM decision if you split the I and Q signal and receive each separately.
Regards,
Ahmad