Hi Mai,
The BER Analyzer approximates the BER by analyzing the eye diagram. Since there is a threshold detector before it the eye diagram is “perfect”, which results in the unrealistically high Q factor. For QAM modulation it would be better to use either the BER Test Set (with a suitably long sequence) or you could use the error vector magnitude metric, which is generally used for QAM signals. This is calculated by the electrical constellation visualizer.
Regards