Dear Karanveer,
Unfortunately, OptiSystem doesn’t support M-ary Duobinary pulse Generator. We’ll consider adding that in the future.
You need to make sure that the M-ary Duobinary pulse generator is placed after the QAM Sequence Generator. That means, the input of the M-ary Duobinary Generator should be M-ary and the output is Electrical.
You could use the Matlab component to make the M-ary Duobinary pulse generator.
Regards,
Ahmad