Hi zulkarnain,
As far as modulation schemes are concerned, there are number of digital modulation schemes you can use for your system design. Please it would be convenient if you try to be more specific regarding your system design. However, i am attaching a link which mentions different digital modulation schemes. You may refer to it for further reference. It would definitely help you.
https://optiwave.com/category/products/system-and-amplifier-design/optisystem/optisystem-applications/advanced-modulation/
regards