Dear Nabila,
please check the example (112Gbps SP-16QAM_B2B_subsystem with DSP_Investigating Power level calculation.osd) located at the following link in OptiSystem Example Library. This example is for 16QAM. However, it can be modified for other QAM levels. I hope this helps you.
When the BER is 0, the system performs perfectly. On the other hand, when BER is 1, the system is causing all bits to be erroneous.
C:\Users\USER NAME\Documents\OptiSystem 17.1 Samples\Advanced modulation systems\QAM systems\16 QAM
Regards,
Ahmad