Hi Yizhou,
As an M-Ary signal there are two main ways. One using a BER Test Set that compares an input sequence to and output sequence exactly, but needs long bit sequences to calculate very small BERs.
Or like in the attached example, using a constellation diagram and a custom target file also attached. I made a PAM4 scheme and a custom target file to calculate the symbol error rate and the error vector magnitude. Make sure before running you correctly define the directory of the .dat file in the Constellation Visualizer properties.
Cheers