Hello Buki,
I recommend you to check the examples at the following locations for guidance on the Matlab signal structures PSK and QAM systems.
C:\Users\USER NAME\Documents\OptiSystem 17.0 Samples\Advanced modulation systems\PSK systems
C:\Users\ USER NAME \Documents\OptiSystem 17.0 Samples\Advanced modulation systems\QAM systems
C:\Users\ USER NAME \Documents\OptiSystem 17.0 Samples\Software interworking\MATLAB co-simulation
Regards,
Ahmad