Dear Sarah,
The fluctuations in OptiSystem is due to the randomness of generated signals in the different components in the software. I agree with you that the BER should get worst when the OSNR decreases, which is not happening in your figure. However, I can’t comment on the effect of OSNR in your system unless I saw the whole project design. You may send me the project as a zip file to check it and provide feedback.
my email is ahmad.atieh@optiwave.com
Regards,
Ahmad