Indeed, the increase in the quadrature demodulator cut-off filter would just stop signal being filtered and therefore provides an output. It is strange that you managed to get results in OptiSystem 7, as the quadrature modulator output was showing signal with virtually zero amplitude.
I have noticed here is also a problem with decision boundaries. Could you modify them first according to your design?