Hello Neha,
Your project layout is confusing. Why you split the signal from the laser, then you multiplex it? You can take it directly to the modulator!
My prediction to the cause of the error is the MUX or DEMUX frequency setup.
You may send me the project file as a zip file to my email at ahmad.atieh@optiwave.com to give you more insight on the project.
Regards,
Ahmad