Dear Amit,
I’ve checked your project and swept the FSO range from 0.1km to 2km. The difference is minor between the two systems. which is expected because the symbols are far from each other and there is no stress on the detection system. However, the PSK requires amplitude/phase information compared to the QAM which requires amplitude only. That makes QAM slightly better in your case.
Regards,
Ahmad