Dear Aisha,
You may use the example (Coherent 4 QAM.osd) for guidance on QAM4 modulated signal transmission. It is located at the following link. You may use direct detection if you require that by replacing the coherent detection part.
C:UsersUSER NAMEDocumentsOptiSystem 19.0 SamplesAdvanced modulation systemsQAM systems
Regards,
Ahmad