Hello Metiz
According to your attached screen shots,system seems correct and i will check your system file too,right now i havnt access to it.In your system you have used LED,but LED has broad spectrum over frequencies,so it is better to use Laser due to coherence and more pointing in the system.According to your second doubt,as you increase 4QAam to 16QAam,level of noises rise and dominate tge signal .It is better to use DSP to compensate the effects .
However if system works for LED then it would be good in terms of cost too .i will try it using your system