Dear Vishal,
what you are saying is reasonable. going up from 4QAM to 16 QAM might deteriorate the system unless some other factors would be adapted like losses, components bandwidths….etc. this is exactly the research concepts.
this is should be your task to have a better system with 16 QAM.
hope that clears,
best of luck
thanks
Mohamed Abouseif