I tried out your project and there was one or two problems I saw. First of all I didn’t get the ordered circular signal at the receiver side. I got a very noisy QAM signal with an almost random distribution. The problem here is the implementation of the Laser source. If you are going to use the measured component then it cannot be run in digital mode, because the input signal is not as simple as a digital signal. You will also need to provide an electrical bias to the input signal and attenuation so that Laser is operating above threshold and below saturation (Max power). Try playing around with the Laser source and look at the optical signal until you can get a cleaner output.
Hope this helps!