Dear Jagan,
pleas use the example DP-8PSK back to back.osd located at the following address in OptiSystem example library as a reference for your project.
Regards,
Ahmad
C:\Users\USER NAME\Documents\OptiSystem 16.1 Samples\Advanced modulation systems\PSK systems\8PSK