Hello Jyoti Sharma. Greetings.
Well here i agree with Burhan. It will be easier to guess your problem if you try to upload your osd file. i think you may be doing something wrong after the Mux. Can you Share the snapshot of your signal waveform after the Mux may be that would help. However the problem which you are facing may be because of the high launching power itself. Try increasing the order of the filter and check out the results. I hope you get your issue resolved.
Regards
Atul Sharma