Hello Nasir for the case of OOK as you attached screen shot above, i would recommend you to use low pass bessel filter electrical after PIN photo detector. Instead of attaching BER visualizer to PRBS and RZ you can use 3R re-generator from receiver library. I would suggest you to first see the eye-diagram just after the transmitter to make sure that it is working fine or not? after that check your system to decrease value of FSO attenuation to minimum level for checking. Can you please tell what sequence length you have considered for OOK because in case of OOK sequence length is not large as in case OFDM.