Hi,
For QAM type modulation, and also including more advanced modulation schemes, it is more appropriate to use a BER Test Set. The BER Analyzer calculates the BER from the eye diagram. However, it expects a binary signal and not an m-ary signal you might be using for you QAM scheme.
If this is not the problem, please send over your project file and I’ll take a look at it.
Cheers!