Hello Tung,
I simulated your system design at i was able to get Quality factor up around 6. As Alistu mentioned above , you can increase laser input power.
I also tried to lessen number of ports at the Demux side. Why are you floating so many ports when they are unused. Make the number of ports as required.
Please check the osd file i have attached Herein. Thanks!