Hello,
I fixed up your project and got it working there was problems with the transmitter!
Be careful when copying components from example files into a new project, because sometimes they will be referencing parameters that might not exist in your layout. For example I had to add a layout parameter called “rolloff” to your project so that the M-ARY Raised Cosine Pulse Generator would give reasonable results.
Moreover, careful consideration of the biasing and driving of MZM’s is needed. I added an electrical gain and bias to the inputs so I could copy close values from the example file. Then I played with the gain until I got good results. It is also possible to modify the gain and bias inside the parameters of the MZM, but I prefer explicitly setting the gain and bias before and only changing the Vpi of the MZM.
Regards