Hi Anita,
I found some design errors in your project file:
1) The channels defined in your WDM Demux do not correspond to the WDM Transmitter channels (1546 nm, 1546.8 nm . . .) I fixed this.
2) The Low pass filters did not have the proper bandwidth, because they were using the Symbol rate (but each channel is a bit stream). I fixed this but I would actually recommend using a Bessel filter here. I think Raised Cosine Filter’s are more popular when working with more advanced modulation schemes.
I’ve attached the project file where I achieved ~2e-8 BER. Note that I am using OptiSystem 13.0, so you might have trouble opening it.