The BER of PAMx and xQAM can be calculated using only the BERT Test Set component. You may use its internal PRBS to feed the binary signal or external source. The BER analyzer can’t be used to monitor the BER or Q-factor of multilevel signal because the process used in the calculation is for binary system as shown in the datasheet (you may access it by right click on the BER Analyzer, choose component properties then Help).
There is new component coming up in the new release 19.0 in couple of months would handle that.
Regards,
Ahmad