Hi Roman,
The BER from the BER Analyzer is calculated in a statistical manner (which it has to be for such small sequence lengths) from the size of the eye opening and the standard deviation of the pulses from the ideal positions. In your case, you have M-ary threshold detectors and psk decoders before the BER analyzer and thus the pulses are now at their ideal positions so there are no standard deviations and therefore zero BER.
So you should either modify your system so that the signal is sent to the BER Analyzer before being pinned at the thresholds or look at our BER test set, this component doesn’t estimate the BER, but does a straight counting, comparing initial signal to final signal. The drawback of the counting method of course is that it may take a very long sequence to actually see a bit error.