Well yeah, that seems to happen especially when more than 4 channels are being used.
I believe changing the bandwidth of the WDM MUX/DEMUX can change the Q.F. of the channels you have an issue with.
Try reducing the bandwidth by a small value and check if anything improves. If an improvement is achieved, then you can decide the final acceptable bandwidth of your MUX/DEMUX.