Dear Roman,
You can calculate the BER in many ways one way is to use the BER test set component that is available in the component library.
the input terminal of BER test set connected to the decoder output or if if you are using a dual polarization scheme them connect to the output of the parallel to serial converter.
you can also use WDM analyzer to check the OSNR.
Given the OSNR, the empirical formula to calculate BER for single fiber is
Log10 (BER) = 10.7-1.45 (OSNR)
Example:
Assume that OSNR = 14.5 dB
ThenLog10 (BER) = 10.7-1.45 (14.5) = -10.30
Therefore BER = 10^(-10.30)
BER is approx 10^-10
I think it helps
Regards