Ok so there are few changes you should make:
1) The Mach-Zehnder Modulator automatically normalizes the electrically input, so the gain and bias components are not really doing anything. The same goes for the gain on the output. However, you can keep them there if you want.
2) The real problem is the Low Pass Filter in the Quadrature Demodulator, you should play around with this value to get better results. I was getting pretty good constellations for a cutoff frequency of 50-100 MHz.
You might also want to create your own demodulator circuit so you can see view the spectrum before the filter. Double click the Quadrature Demodulator and look at the help file to see how it is built.
Hope this helps!