Hi Gokul,
QAM modulation using Matlab and without OptiSystem (what you have asked for above) is pretty simple. You would just have to use “qammod” function for modulation and “qamdemod” for the demodulation of the signal for a given binary vector (which is supposed to be you data vector).
Regards